---
title: Relative Position Encoding
url: https://www.emergentmind.com/topics/relative-position-encoding
type: topic
---

# Relative Position Encoding

Relative Position Encoding (RPE) is a class of positional encoding schemes for sequence, image, graph, and multi-modal transformer models in which positional information is injected not as a function of token indices themselves, but as a function of the **relative distance or relation** between pairs of tokens, nodes, or elements. In contrast to absolute position encoding—which ties model representations to fixed coordinate systems—RPE enables translation-invariant or shift-invariant modeling, better generalization to longer or differently-structured inputs, and more direct encoding of local and non-local relationships. RPE schemes have become foundational in large language models, vision transformers, audio, graph, and multi-view/multi-modal transformers.

## 1. Principles and Mathematical Foundations

Classic transformer models, such as in Vaswani et al. ("Attention Is All You Need"), inject absolute position embeddings $p_i$ into each input token $x_i$, yielding content that is sensitive to the single position index $i$. In contrast, relative position encoding injects pairwise information $r_{i-j}$ (or variants thereof) in the attention mechanism so that attention weights, or their parameterizations, become functions of the **distance $i-j$** (or a general relational operator) between tokens or elements.

A canonical mathematical scheme for 1D sequences is the so-called "key-relative bias" variant:

\[
\mathrm{Attention}(q_i, k_j) = \frac{1}{\sqrt{d}} \big[ q_i^\top (k_j + r_{i-j}) \big]
\]
with $r$ as a learned or structured table of bias vectors indexed by offset $i-j$ [2005.09940][2009.13658][2109.02377].

Alternative forms include direct bias addition to logits ("attention-bias"):

\[
\mathrm{Attention}(q_i, k_j) = \frac{1}{\sqrt{d}} \left( q_i^\top k_j + b_{i-j} \right)
\]
with $b$ a learned or structured bias vector [2501.16745][2305.16642][2407.02934].

Empirically, relative encodings:
- Generalize out-of-domain to longer sequence lengths (since they store only offsetwise, not absolute, relations).
- Require fewer parameters to capture spatial or sequential relations ($O(L)$ or $O(2L-1)$) versus explicit absolute tables over potentially unbounded position indices.
- Better support translation invariance, topology-awareness (graph, 2D, or 3D geometry), and can reflect locality or directionality.

## 2. Core Methods Across Modalities

### Sequences and Language
**Shaw et al. (2018)-style** RPE adds learned embedding vectors $r_{i-j}$ or $b_{i-j}$ to keys or directly as attention biases, with distance clipping for parameter efficiency [2005.09940][2009.13658]. Enhanced forms generalize the bias to closures over queries and keys (“contextual mode”):

\[
e_{ij} = (Q_i K_j) + (Q_i r_{i-j}) + (K_j r_{i-j})
\]
[2009.13658].

**Rotary Position Embedding (RoPE)** represents positions as complex/real plane or Lie group rotations such that $(q_i, k_j)$ rotations encode *only* the difference $j-i$, and generalizations to multidimensional spaces or noncommutative rotations are possible [2406.10322][2109.02377].

**Efficient RPE for Linear Attention:** For kernel-based or Performer attention ($O(L)$ time), naive pairwise RPE breaks tractability. Solutions such as PermuteFormer encode relative position by position-dependent linear maps (permutations, scalings), preserving relative invariance and linear complexity [2109.02377][2307.09270].

### Vision (2D, 3D, Multimodal)
**2D and 3D Scheme Extension:** 
- *Directional and cross-product RPEs* separate x and y (and z) offsets, with contextual bucketing by (Δx, Δy) or their products; this enables explicit modeling of horizontal, vertical, or spatially composite relations [2107.14222][2308.04409].
- *Contextual/semantic-aware RPE* (e.g., SaPE²) leverages gates or affinity scores informed by content, yielding content-aware RPE that tracks semantic similarity rather than just geometric distances [2505.09466].
- *Affine-invariant RPE* (e.g., KP-RPE) adapts bias fields based on keypoint anchors, robustly encoding relationships under affine image transformations [2403.14852].

**Multi-view, Camera, and Multi-modal Relative Encoding:**
- Camera-aware RPEs (PRoPE) encode the full projective geometry (intrinsics and extrinsics) between viewpoints in the attention bias, supporting robust generalization under camera/scene variation [2507.10496].

### Graphs and Structured Domains
**GRPE** introduces relative position encodings on graphs using topological distances (shortest path) and edge types, integrated as content-aware vector dot products between node queries/keys and tabled relation embeddings. This scheme avoids graph linearization and preserves full topology-awareness [2201.12787].

### Temporal and Spatio-Temporal
**Video and Time Series RPE:** For videos and temporal signals, RPE is extended to temporal, spatial, or spatio-temporal axes, often as low-rank or parameter-efficient dictionaries indexed by relative offsets. These are efficiently combined in grouped or factorized MLP or sparse attention blocks [2407.02934][2305.16642].

## 3. Theoretical Properties and Algorithmic Trade-offs

Relative position encodings are fundamentally defined by their **shift, translation, and topology invariance**:
- *Shift-invariance:* Attention scores depend only on $i-j$, so models trained on short sequences can immediately extrapolate.
- *Parameter efficiency:* Only $O(L)$ parameters or fewer, especially with bucketing or tying, even in high-dimensional settings.
- *Algorithmic overhead:* Standard RPE increases $O(L^2)$ cost, breaking the linear-time properties of efficient transformer variants unless specially designed (e.g., via unitary transforms, permutations, or kernel compatible RPE) [2109.02377][2307.09270].

**Empirical findings** indicate that:
- Contextual and cross-content RPEs outperform simple bias variants in vision/classification.
- The inductive bias introduced (e.g., by directional or semantic bucketing) improves structure-sensitive domains (images, graphs, code).
- RPE can fully replace absolute position encoding in many domains with no loss and frequently with measurable gain.
  
A summary of variants and their compatibility/complexity properties:

| Scheme                    | Complexity | Parameter Count        | Generalization      |
|---------------------------|------------|------------------------|---------------------|
| Shaw et al. 1D/2D         | $O(L^2 d)$ | $O(2k d)$ or $O(k^2 d)$| Good (distance)     |
| PermuteFormer             | $O(L m^2)$ | negligible extra       | Full (linear time)  |
| LRPE/Unitary (linear attn)| $O(L d^2)$ | $O(d^2)$ per pos       | As above            |
| Contextual (product/cross)| $O(n k d)$ | $O(k d)$               | Enhanced (semantics)|
| Semantic-aware (SaPE²)    | $O(N^2)$   | $O(M+1)$               | Semantic + shift-inv|
| Graph (GRPE)              | $O(N^2)$   | $O(L d)$               | Full (graph)        |

## 4. Domain-Specific Adaptations and Innovations

- **Speech:** RPEs built from bidirectional sinusoids adapt seamlessly to much longer and more variable input with negligible extra cost, outperforming absolute encoding on multi-hour speech recognition and translation [2005.09940][2106.03143].
- **3D Object Detection:** Vertex-based RPE encodes per-query, per-vertex offsets in canonical boxes, providing differentiable, locality-aware priors that outperform both standard RPE and hard box-masks in DETR-style architectures [2308.04409].
- **Multi-modal Models:** Circle-RoPE addresses cross-modal bias in large vision-language models, mapping image token positions onto a geometric structure orthogonal to text, reducing artificial alignment and improving spatial robustness [2505.16416].
- **Affine Robustness in Vision:** KP-RPE achieves alignment invariance by warping positional bias fields conditional on detected keypoints, preserving spatial priors under large transformations [2403.14852].

## 5. Empirical Impact, Ablation Studies, and Limitations

RPE forms consistently deliver gains across vision (up to +1.5% Top-1 on ImageNet in DeiT/ViT), speech (up to 7% relative WER reduction), code-editing (1–3% absolute improvement in patch accuracy), and time series (best average rank across 30+ multivariate datasets). Ablations confirm:
- Contextual and multi-branch (Q, K, V) injections are superior to scalar or index-only biases [2107.14222][2009.13658].
- Composite directional buckets and semantic gate mechanisms enhance translation and scale equivariance.
- For efficient attention (Performer, linear transformers), only specially designed RPEs (e.g., PermuteFormer, linearized unitary transforms) can maintain $O(L)$ complexity.

Limitations arise due to increased $O(N^2)$ cost in naive high-dimensional or semantic RPE, limitations in encoding beyond permutation order in permutation-based methods, and sensitivity to domain-specific preprocessing (e.g., keypoint detection, camera pose consistency). Open challenges include further reducing memory/computational overhead for very large contexts and enabling differentiable learning of permutation or semantic structure in efficient RPEs [2109.02377][2505.09466][2305.16642].

## 6. Future Directions and Open Challenges

Research directions for RPE include:
- **Learning or adapting permutation/order structure** within efficient attention kernels to extend expressivity and distance range [2109.02377][2307.09270].
- **Integrating semantic or task-specific cues** (e.g., keypoints, semantic grouping, topology) for robust transfer across modalities and domains [2505.09466][2403.14852].
- **Hybrid absolute/relative encoding** to combine out-of-domain scale robustness with fine-grained local structure [2505.09466][2106.03143].
- **Sparse and blockwise RPE** for scalability in dense or high-resolution spatio-temporal problems [2407.02934].
- **Lie group-based and geometry-aware extensions** (LieRE, geo-RoPE, Camera-based) for generalizing beyond regular grids or to data on manifolds or with rich geometric structure [2406.10322][2403.15940][2507.10496].
- **Graph/molecule and code-specific RPE** integrating neighbourhood topology, edge types, or parse-tree substructure [2201.12787][2205.13522].

## 7. Key References

- "PermuteFormer: Efficient Relative Position Encoding for Long Sequences" [2109.02377]
- "Relative Positional Encoding for Speech Recognition and Direct Translation" [2005.09940]
- "A 2D Semantic-Aware Position Encoding for Vision Transformers" [2505.09466]
- "Toward Relative Positional Encoding in Spiking Transformers" [2501.16745]
- "Rethinking and Improving Relative Position Encoding for Vision Transformer" [2107.14222]
- "V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection" [2308.04409]
- "GRPE: Relative Positional Encoding for Graph Transformer" [2201.12787]
- "Improve Transformer Models with Better Relative Position Embeddings" [2009.13658]
- "Linearized Relative Positional Encoding" [2307.09270]
- "Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models" [2505.16416]
- "PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition" [2407.02934]
- "LieRE: Lie Rotational Positional Encodings" [2406.10322]
- "HyPE: Attention with Hyperbolic Biases for Relative Positional Encoding" [2310.19676]
- "CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings" [2106.03143]
- "Geotokens and Geotransformers" [2403.15940]
- "KeyPoint Relative Position Encoding for Face Recognition" [2403.14852]

Relative position encoding has thus evolved into a foundational, extensible mechanism for geometric, relational, or context-invariant modeling across modern transformer architectures.

Source: https://www.emergentmind.com/topics/relative-position-encoding