---
title: Relative Positional Encodings in Transformers
url: https://www.emergentmind.com/topics/relative-positional-encodings
type: topic
---

# Relative Positional Encodings in Transformers

Relative positional encodings are patterns, mechanisms, or parameterizations used within Transformer models to supply positional information in a form that is invariant (or equivariant) under translation of the input, so that the model’s core operations depend only on the relative arrangement of tokens rather than their absolute sequence indices. This class of encodings underpins much of the recent progress in scaling, expressivity, and generalization of Transformers across natural language, vision, audio, and multimodal architectures. Relative encodings subsume a variety of concrete schemes, including formalisms based on Lie groups, explicit Toeplitz kernels, low-rank decompositions, learned trainable tables, and geometric invariants, each with their own mathematical and statistical properties.

## 1. Mathematical Foundations and General Principles

The fundamental distinction of a relative positional encoding (RPE) is its dependence on *differences* of position indices (or more generally, coordinates) rather than on absolute position. For sequential data, if $i$ and $j$ denote token positions (or coordinates $x_i, x_j$), the RPE intrinsically references $j-i$ or $x_j - x_i$. In the canonical form, the attention logit $e_{ij}$ in a single Transformer head is augmented as
\[
e_{ij} = (q_i^\top k_j) + f(j-i),
\]
where $f(\cdot)$ is a (possibly learned or fixed) bias function of relative offset. For graph and multiview data, more general structures (such as shortest-path, resistance distance, or inter-camera projective transforms) can play the role of $f$.

Several architectures further multiply or rotate the query/key vectors with position-dependent operators $R(i), R(j)$, so that the attention logit is
\[
e_{ij} = (q_i R(i))^\top (k_j R(j)) = q_i^\top (R(i)^\top R(j)) k_j,
\]
with $R(i)^\top R(j)$ depending only on $(j-i)$ in the appropriate group structure. This form captures all group-action–based relative encodings (LieRE, RoPE, GRAPE, etc.) [2406.10322, 2512.07805]. The group property $R(i)^\top R(j) = R(j-i)$ ensures translations or equivariant transformations do not alter the relative attention pattern, which is crucial for invariance and compositionality.

For general attention kernels $K(i, j)$, RPEs often require the existence of a decomposition such that $K(i, j) = h(i-j)$ or $K(i, j) = h(x_i - x_j)$ for some function $h$.

## 2. Taxonomy of Relative Positional Encoding Schemes

Relative encodings encompass a broad spectrum of parameterizations:

- **Additive bias tables**: Explicitly learn or fix a vector or scalar for each relative offset $k$ (Shaw et al. [2009.13658], T5, ALiBi [2506.06398, 2401.16421], KERPLE [2401.09686]).
- **Multiplicative/group action schemes**: Rotate or otherwise transform query and key vectors by group elements parameterized by position (RoPE [2505.13027], LieRE [2406.10322], GRAPE-M [2512.07805]).
- **Low-rank or kernel approximations for linear Transformers**: Realize RPEs via product of position-dependent feature maps, e.g., stochastic positional encoding (SPE) [2105.08399], linearized RPE (LRPE) [2307.09270].
- **Toeplitz or spectral bias kernels**: Model $f$ as a function inducing a Toeplitz matrix and study its spectral properties for expressivity/stability [2505.13027].
- **Geometric/graph encodings**: Use graph-theoretic distances, resistance, magnetic Laplacian embeddings, or camera frustum relations for relative encoding over more general data domains [2402.14202, 2407.20912, 2507.10496].
- **Binary codes and quantized schemes for spiking/binary SNNs**: Employ Gray code–based or logarithmic code RPEs compatible with binary network operations [2501.16745].
- **Hybrid approaches**: Blend segment-level absolute and inter-segment relative encoding (BiPE [2401.16421]), or mix identity and rotary streams as in MLA [2505.13027].

A technical table below summarizes core classes:

| Scheme                   | Parameterization Domain | Notable Property         |
|--------------------------|------------------------|--------------------------|
| Additive bias (Shaw/ALiBi)| $\mathbb{R}$ or $\mathbb{R}^{d}$ per offset | Linear/learned bias, streaming-friendly |
| Multiplicative group (RoPE, LieRE, GRAPE) | Lie group (SO(d)), matrices | High-variety, group-theoretic compositionality |
| Kernel/spectral (Toeplitz, wavelet) | Operators, Toeplitz/spectral | Spectral contraction, condition number control |
| Linearized (SPE, LRPE)   | Low-rank product space | $\mathcal{O}(n)$ compute and memory|
| Geometric/graph-wise     | Graph metrics, $SE(3)$, projective groups | Data-dependent invariants, heterogeneous domains|

## 3. Theoretical Properties: Expressivity, Generalization, Extrapolation

Relative positional encodings fundamentally alter the expressivity and generalization properties of transformer models. The expressivity of an RPE-equipped transformer is characterized by its ability to represent invariant functions of input that depend only on pairwise or groupwise relative positional relations, not absolute sequence position. Formal results articulate that:

- The class of functions implementable by an $L$-layer, $H$-head transformer with relative PE is dense in the set of continuous functions invariant under position translation [2506.06398].
- Pure RPE models cannot represent tasks that require absolute position awareness (e.g., “is token $i$ in the first 10%?”), while any function of relative offset $f(x_j,x_i, i-j)$ is fully expressible [2506.06398].
- For generalization, Rademacher complexity bounds show that regularization/decay on the parameter norms of the offset bias or generator controls overfitting and yields uniform convergence rates comparable to classic absolute encodings [2506.06398].
- Extrapolation to sequence lengths $N > N_{\text{train}}$ is strictly impossible for learned absolute and compactly parameterized RPEs (with a hard cutoff/clipping at $|i-j|>K$), but is well-supported by ALiBi (linear slope), HyPE (hyperbolic) [2310.19676], and kernelic/spectral bases [2506.06398, 2505.13027], as their functional forms remain well-defined for arbitrary $i, j$ [2401.16421].

## 4. Implementation Methodologies in Transformer Architectures

RPEs are integrated into Transformers at various algorithmic locations:

- **Key-query logit modification**: Augment the dot-product $q_i^\top k_j$ with $f(j-i)$ or more generally $f(q_i, k_j, j-i)$. This is directly implemented in the additive and kernel/spectral schemes [2009.13658, 2407.20912].
- **Query/key transformation**: Pre-multiply $q_i, k_j$ by $R(i), R(j)$ as in RoPE, LieRE, GRAPE, so that $e_{ij}$ depends on $q_i^\top R(j-i) k_j$ [2406.10322, 2512.07805, 2505.13027].
- **Attention kernel design for linear transformers**: Constrain $K(i, j)$ to have a decomposition $h(q_i, i)^\top h(k_j, j)$, which permits low-memory, fast-transform methods [2105.08399, 2307.09270].
- **Graph Transformer bias injection**: For general data, a precomputed matrix $B_{ij} = f(\mathrm{SP}(i, j))$ where $\mathrm{SP}(i, j)$ is shortest path or other graph metric, is added to the attention logits [2402.14202].
- **Specialized domain encodings**: In computer vision or 3D perception, geometric relation matrices (projective, $SE(3)$, etc.) transform token features or attention kernels to render them invariant to camera pose, scene rearrangement, or projection [2507.10496].

Efficient implementations exploit blockwise decomposition (e.g., $2\times2$ blocks in rotary/GRAPE), precompute transformations for repeated positions, use streaming caches for autoregression, and exploit structure (Toeplitz, group action) for batch suitability.

## 5. Empirical Evaluation and Domain-Specific Impact

Relative positional encoding schemes achieve state-of-the-art results across a spectrum of tasks and modalities:

- **Vision**: LieRE outperforms RoPE-Mixed and absolute positional encoding on CIFAR100 (+2.7% over RoPE-Mixed, +5.5% over absolute) and UCF101/RSNA/3D tasks (+2.5% to +6.7% gains), with enhanced data/compute efficiency and robustness to patch shuffling [2406.10322].
- **Speech and audio**: T5-style and kernel RPEs (KERPLE) yield persistent improvements in PESQ/ESTOI across SNRs and outperform both fixed and learned absolute embeddings in noncausal Transformers [2401.09686]. Relative encoding in speech Transformers boosts WER/BLEU robustness under segmentation quality shifts, exceeding absolute PE in both ASR and speech-to-text translation [2005.09940].
- **Language modeling and QA**: RPEs generalizing both query- and key-relative interactions (including full three-way dot product forms) deliver up to 2 F1 improvements on SQuAD and maintain accuracy at extended sequence lengths [2009.13658]. Wavelet-based RPEs have leading extrapolation on synthetic long-context tasks [2506.06398].
- **Graph learning**: Multi-$q$ Magnetic Laplacian PEs in directed graphs permit full recovery of walk profiles; empirically, they reduce RMSE in distance/walk-prediction by up to 70% over Laplacian or SVD-based encodings, and substantially outperform single-$q$ and random walk baselines on circuit, sorting, and synthetic tasks [2407.20912].
- **Multi-view vision**: Projective Positional Encoding (PRoPE), invariant to both SE(3) and camera intrinsics, yields highest PSNR/LPIPS/SSIM on novel view synthesis, maintains robustness under out-of-distribution focal lengths, and improves discriminative spatial cognition in geometric tasks [2507.10496].

## 6. Design Insights, Limitations, and Future Directions

The structure and choice of relative positional encoding are shaped by practical, architectural, and theoretical considerations:

- **Spectral contraction**: Multiplicative Toeplitz-based schemes (e.g., RoPE, LieRE, GRAPE) produce better-conditioned logit matrices than additive-only or bias-based designs, which accelerates and stabilizes optimization [2505.13027].
- **Group-theoretic structure**: Encodings founded on Lie group actions guarantee exact or asymptotic dependence of attention solely on relative position, with higher representational capacity unlocked by enriching the dimension and noncommutativity of acting groups (as in LieRE and general GRAPE).
- **Regularization and expressivity**: For learned RPE tables, norm regularization is vital to prevent overfitting (especially in regime of limited data/long-sequence extrapolation) [2506.06398].
- **Domain-specific demands**: Tasks requiring precise retrieval or copying from arbitrary context make strict long-range decay in the encoding (as in ALiBi) suboptimal; high-frequency or non-decaying encodings (HoPE) can improve both extrapolation and context-awareness [2410.21216].
- **Streaming and compatibility**: RPEs that admit efficient incremental computation and cacheability (e.g., additive GRAPE, ALiBi) are suited to large-scale, streaming, or autoregressive deployment.
- **Hybridization and composition**: BiPE and related schemes that blend absolute and relative encodings (intra vs. inter-segment) empirically and theoretically offer improved performance, especially in extrapolation [2401.16421].

Observable limitations include the inability of RPEs to model purely absolute position-dependent phenomena, finite-capacity learned tables failing on extreme extrapolation, and for some non-group-theoretic methods, an increase in memory or computational overhead without a corresponding gain in performance.

## 7. Domain Extensions and Specialized Architectures

Relative encoding principles admit direct extension to a variety of advanced settings:

- **Binary and spiking networks**: Gray-PE and Log-PE inject positional information compatible with binary SNN hardware, supporting time-series, text, and vision tasks with negligible computational overhead [2501.16745].
- **Graph and geometric data**: Multivariate, permutation-equivariant relative encodings via shortest path, resistance, spectral, or projective transformations permit Transformers to operate on irregular geometries, directed networks, and multi-view data, achieving close to best-known accuracies on circuit-, vision-, and program-analysis benchmarks [2407.20912, 2507.10496].
- **Linear-Complexity Transformers**: Structured decompositions (LRPE, SPE) allow full relative encoding even when strict $\mathcal{O}(n)$ complexity is imposed, as needed for single-pass multi-thousand token contexts [2105.08399, 2307.09270].

The design space for RPEs is thus substantial and expanding, with active research mapping the trade-offs between expressivity, computation, extrapolation, and robustness required by evolving applications.

---

**References**  
- [2406.10322] LieRE: Lie Rotational Positional Encodings  
- [2512.07805] Group Representational Position Encoding  
- [2505.13027] Unpacking Positional Encoding in Transformers: A Spectral Analysis of Content-Position Coupling  
- [2506.06398] Theoretical Analysis of Positional Encodings in Transformer Models  
- [2009.13658] Improve Transformer Models with Better Relative Position Embeddings  
- [2401.16421] Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation  
- [2005.09940] Relative Positional Encoding for Speech Recognition and Direct Translation  
- [2401.09686] An Empirical Study on the Impact of Positional Encoding in Transformer-based Monaural Speech Enhancement  
- [2501.16745] Toward Relative Positional Encoding in Spiking Transformers  
- [2410.21216] HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation  
- [2307.09270] Linearized Relative Positional Encoding  
- [2105.08399] Relative Positional Encoding for Transformers with Linear Complexity  
- [2507.10496] Cameras as Relative Positional Encoding  
- [2407.20912] What Are Good Positional Encodings for Directed Graphs?  
- [2402.14202] Comparing Graph Transformers via Positional Encodings

Source: https://www.emergentmind.com/topics/relative-positional-encodings