---
title: 2D Rotary Positional Embedding for Transformers
url: https://www.emergentmind.com/topics/2d-rotary-positional-embedding-rope
type: topic
---

# 2D Rotary Positional Embedding for Transformers

A 2D Rotary Positional Embedding (RoPE) is a positional encoding mechanism for Transformer architectures that injects relative two-dimensional location information into token representations via axis-wise or mixed-frequency rotations. Developed to generalize the block-diagonal rotation of 1D RoPE—well-established in language models—to vision and multimodal domains, 2D RoPE enables precise and efficient spatial encoding, systematic extrapolation to unseen resolutions, and seamless integration into multi-head self-attention modules. The construction of 2D RoPE is grounded in Lie-theoretic principles, ensures relativity and reversibility, and is supported by both empirical and mathematical research across visual transformers, multimodal systems, agent modeling, and robotic perception.

## 1. Theoretical Foundations and Properties

2D RoPE is formalized through the lens of Lie group and Lie algebra theory, providing a principled basis for rotational positional encoding in higher (e.g., 2D, 3D) input spaces. Two core properties are central to all valid 2D RoPE constructs:
- **Relativity:** For all positions $(i_1, j_1)$ and $(i_2, j_2)$, the rotation matrices satisfy $R_{(i_1, j_1)}^\top R_{(i_2, j_2)} = R_{(i_2-i_1, j_2-j_1)}$. This ensures that the query-key attention scores depend only on relative spatial offsets, not absolute positions.
- **Reversibility (Injectivity):** $R_{(i_1, j_1)} = R_{(i_2, j_2)}$ only if $(i_1, j_1) = (i_2, j_2)$, preserving unique encodings for unique positions [2504.06308].

These properties require that, within the $\mathfrak{so}(4)$ Lie algebra, the generators $B_1$ and $B_2$ associated with the $x$ and $y$ axes must commute and span a maximal abelian subalgebra (MASA). The canonical instantiation block-diagonalizes into two independent $2 \times 2$ rotation planes; more expressive inter-axis coupling is achieved by a learned orthogonal basis transformation $Q$, preserving all group-theoretic properties [2504.06308, 2406.10322, 2502.02562].

## 2. Mathematical Formulation

Let a query/key embedding have head dimension $d$, divisible by four. Typical 2D RoPE constructs the positional rotation matrix as follows:

- **Axial 2D–RoPE (Block-Diagonal, “Pure”):** Divide the $d/2$ complex channels between $x$ and $y$. For $t = 0, \dotsc, d/4 - 1$,
  \[
    R_{axial}(n)_{2t} = e^{i \theta_t p^x_n}, \quad R_{axial}(n)_{2t+1} = e^{i \theta_t p^y_n}
  \]
  with frequencies $\theta_t = 100^{-t/(d/4)}$ [2403.13298, 2504.06308].

- **Mixed-Frequency 2D–RoPE (RoPE-Mixed):** Introduce learnable per-head frequency vectors $\theta^x, \theta^y \in \mathbb{R}^{d/2}$,
  \[
    R_{mixed}(n)_t = \exp\left[i(\theta_t^x p_n^x + \theta_t^y p_n^y)\right]
  \]
  This variant enables encoding of all possible offset directions (including diagonals) [2403.13298].

Both variations ultimately perform an elementwise complex (or real-valued) multiplication of each query/key embedding with the rotation for its $(x, y)$ location, transforming $q_n$, $k_m$ into $q'_n = q_n \circ R(n)$, $k'_m = k_m \circ R(m)$, and computing attention scores via $\operatorname{Re}\left[q'_n k'^{*}_m\right]$.

In the Lie-theoretic formalism, the most general 2D RoPE is expressed as
\[
  R_{(i, j)} = \exp(i B_1 + j B_2), \quad [B_1, B_2] = 0 
\]
where $B_1$ and $B_2$ are commuting $4 \times 4$ skew-symmetric generators, potentially transformed as $Q\,\mathrm{diag}(J(i\theta), 0)\,Q^\top$ and $Q\,\mathrm{diag}(0, J(j\theta))\,Q^\top$ to model axis interaction [2504.06308, 2502.02562]. 

The infinitesimal generators can be arranged as block-diagonal matrices to yield a rotation of arbitrary axis, block, and frequency allocation across the embedding [2406.10322, 2502.02562].

## 3. Integration in Vision Transformer Architectures

2D RoPE directly replaces or augments the positional encoding step in Vision Transformers (ViT), Swin Transformers, and related image or spatially-structured models:
- **Indexing:** Tokens are assigned 2D coordinates $(i, j)$ on a grid (typically flattening in row-major order, $n = iW + j$).
- **Embeddings:** Each token's query/key is rotated according to its $(i, j)$ position and the chosen frequencies.
- **Attention:** Attention computation proceeds using dot products of the rotated queries and keys, guaranteeing that attention weights are modulated by relative spatial offsets [2403.13298, 2505.06991].

In practical implementations, rotation matrices are cached or constructed per head/layer/resolution, and the per-token cost is negligible ($<0.1\%$ extra FLOPs, $<0.01\%$ additional parameters for ViT-S/B). The RoPE module does not mandate changes to the backbone training pipeline and is compatible with dense prediction, segmentation, detection, and windowed attention [2403.13298, 2505.06991].

## 4. Generalizations and Modal Extensions

The explicit Lie-algebraic construction admits further generalization and flexible adaptation:
- **STRING:** Provides universal, separable, translationally invariant PEs for $d_c$-dimensional coordinates, showing that all block-diagonal RoPEs are special cases of exponential maps generated by commuting skew-symmetric matrices [2502.02562].
- **LieRE:** Parametrizes and learns more general (potentially non-separable) 2D/3D rotations, achieving higher capacity and extension to arbitrary input dimensionality; block structure is employed for computational efficiency [2406.10322].
- **Directional RoPE (DRoPE):** Extends 2D RoPE to encode agent heading via a $2\pi$-periodic block-diagonal, crucial for maintaining invariance under angular wrap-around in trajectory modeling [2503.15029].
- **Spherical RoPE:** Adapts RoPE to spherical coordinates by constructing $3 \times 3$ blocks corresponding to latitude and longitude, avoiding frequency scaling as angles are taken in natural units (radians), with utility in geographic transformer architectures ("geotokens") [2403.15940].
- **VideoRoPE:** Further extends the block-diagonal concept to 3D (spatio-temporal) settings, carefully allocating frequency bands to spatial and temporal axes, introducing diagonal layout and adjustable temporal spacing. The 2D special case remains directly compatible for image-only models [2502.05173].

## 5. Empirical Performance and Ablation Evidence

Quantitative evaluations establish that 2D RoPE consistently outperforms absolute sine/cosine PEs, learnable absolute PEs, and additive/relative bias in diverse vision and dense prediction tasks:

| Model/Task     | Baseline PE         | RoPE-Mixed         | Gain                 |
|----------------|---------------------|--------------------|----------------------|
| ViT-B/224, I1k | APE = 83.4%         | 83.8% (+0.4 pp)    |                      |
| Swin-B/224, I1k| RPB = 83.3%         | 83.7% (+0.4 pp)    |                      |
| COCO/AP, ViT-B | APE = 49.4          | 51.2 (+1.8 pp)     |                      |
| ADE20k/Vit-B   | APE = 47.7          | 49.6 (+1.9 pp)     |                      |
| ADE20k/Swin-S  | RPB = 50.2          | 51.1 (+0.9 pp)     |                      |

These improvements persist under distribution shift and unseen input resolutions due to RoPE's translation and extrapolation guarantees [2403.13298]. Empirical results from the GOOSE segmentation challenge show $+0.71$ pp mIoU improvement due to RoPE alone [2505.06991]. Ablations indicate that RoPE-Mixed (with learnable frequency vectors) offers systematic advantages over both axial variants and standard additive biases [2403.13298, 2406.10322].

Data- and compute-efficiency are also observed: LieRE attains similar accuracy levels as absolute PE baselines in approximately $3.5\times$ fewer training steps, and the marginal computational overhead per forward pass remains negligible in all tested settings [2406.10322].

## 6. Practical Recommendations and Implementation Trade-offs

Best practices supported by systematic study include:
- Default to RoPE-Mixed, as it delivers the strongest classification, detection, and segmentation performance across multiple ViT and Swin backbones [2403.13298].
- Store per-head, per-layer frequency vectors and recompute the rotation matrices only when the grid shape changes; this amortizes cost.
- Optionally add an absolute PE (APE) when the primary deployment regime is close to the training distribution, at the cost of some out-of-distribution extrapolation.
- For sequence lengths or resolutions well outside the training regime, leverage RoPE's inherent extrapolation capability without retraining or fine-tuning [2403.13298].

## 7. Theoretical Guarantees and Universality

The mathematical justification for RoPE's design space is established via universality results: every continuously differentiable, translationally invariant positional encoding into an orthogonal group arises as
\[
  R(r) = \exp\left(\sum_{k=1}^{d_c} L_k [r]_k\right)
\]
for commuting skew-symmetric $L_k$. The block-diagonal construction of 2D RoPE is thus not only computationally optimal but also a maximally expressive class of translationally invariant position encodings under reasonable smoothness conditions. This underpins the robustness, flexibility, and generality observed empirically and makes RoPE an extensive foundation for further research in high-dimensional and structured positional encoding [2502.02562, 2504.06308].

---

**References:**
- "Rotary Position Embedding for Vision Transformer" [2403.13298]
- "Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding" [2504.06308]
- "LieRE: Lie Rotational Positional Encodings" [2406.10322]
- "Learning the RoPEs: Better 2D and 3D Position Encodings with STRING" [2502.02562]
- "Technical Report for ICRA 2025 GOOSE 2D Semantic Segmentation Challenge: Leveraging Color Shift Correction, RoPE-Swin Backbone, and Quantile-based Label Denoising Strategy" [2505.06991]
- "VideoRoPE: What Makes for Good Video Rotary Position Embedding?" [2502.05173]
- "DRoPE: Directional Rotary Position Embedding for Efficient Agent Interaction Modeling" [2503.15029]
- "Geotokens and Geotransformers" [2403.15940]

Source: https://www.emergentmind.com/topics/2d-rotary-positional-embedding-rope