2D Rotary Position Embedding (RoPE)

Updated 6 August 2025

2D RoPE is a positional encoding scheme that extends 1D rotary embeddings to two spatial dimensions using dimension-wise rotations.
It employs block-diagonal orthogonal matrices and sine-cosine multiplications to embed absolute positions as relative information efficiently.
2D RoPE enhances performance in vision and speech models by improving resolution extrapolation and reducing computational overhead.

A 2D Rotary Position Embedding (RoPE) is a position encoding scheme for transformers and self-attention architectures, extending the original (1D) RoPE concept to two spatial dimensions. It replaces traditional additive encoding by embedding absolute position information via dimension-wise rotations, resulting in implicit, translation-consistent relative position encoding in attention modules. This section surveys the foundations, algebraic construction, computational properties, applications in vision and speech, and extrapolation behaviors of 2D RoPE, as described in foundational and recent literature.

1. Fundamental Principles and Mathematical Formulation

The core of 2D RoPE is the use of specially structured, block-diagonal orthogonal matrices that apply a phase rotation to each embedding subspace, parameterized by spatial position. For a feature vector $x_n$ at position $n$ and spatial coordinate $p_n = (x_n, y_n)$ , the rotary transformation is applied as follows:

Each embedding channel pair (for a $d$ -dimensional vector, $d/2$ pairs) is regarded as a 2D subspace.
In the simplest (1D) form, for position $m$ , the transformation is $R_m = \mathrm{diag}(R_{m\theta_1},... ,R_{m\theta_{d/2}})$ , where each $R_{m\theta_i}$ is the $2\times2$ rotation:

$R_{m\theta_i} = \begin{bmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \ \sin(m\theta_i) & \cos(m\theta_i) \end{bmatrix}$

with $\theta_i = 10000^{-2(i-1)/d}$ (Su et al., 2021).

For 2D inputs such as images, rotary position is parameterized over two coordinates. The transformation can be defined in several forms:

Axial RoPE: Dedicated half-channels to each axis, such that for $p_n = (x, y)$ ,

$R(n, 2t) = \exp[i\,\theta_t\,x], \qquad R(n, 2t+1) = \exp[i\,\theta_t\,y]$

(Heo et al., 20 Mar 2024).

Mixed-Frequency RoPE: Each dimension pair may receive a rotation by a linear combination $\theta_t^x x + \theta_t^y y$ (Heo et al., 20 Mar 2024).

For queries and keys at positions $p_m$ and $p_n$ , the inner product becomes:

$A'_{m, n} = \mathrm{Re}[q_m\, k_n^* \cdot \exp{i(\theta^x (x_m-x_n) + \theta^y (y_m-y_n))}]$

RoPE’s essential property is that the attention dot-product after rotary transformation depends only on the relative position. For 2D, the dot product computes as $q^\mathrm{T} R_{p_n-p_m} k$ , unifying absolute and relative encodings via algebraic structure (Su et al., 2021, Liu et al., 7 Apr 2025).

2. Algebraic and Geometric Foundations

From a Lie algebraic perspective, a general $N$ -dimensional RoPE is formalized as:

$R(\vec{x}) = \exp\Big( \sum_{i=1}^{N} x^i B_i \Big)$

where $B_i$ are commuting skew-symmetric generators (elements of a maximal Abelian subalgebra, MASA, of $\mathfrak{so}(d)$ ). Relativity— $R(\vec{x}_1)^\mathrm{T} R(\vec{x}_2) = R(\vec{x}_2 - \vec{x}_1)$ —and injectivity (reversibility) constrain the form of the transformation (Liu et al., 7 Apr 2025). The block-diagonal (axial) construction is maximally efficient but separable across axes; richer variants (e.g., mixing axes or introducing a learned orthogonal basis) can capture richer geometric relations, including diagonals and inter-dimensional coupling (Heo et al., 20 Mar 2024, Schenck et al., 4 Feb 2025, Liu et al., 7 Apr 2025).

3. Computational Properties and Complexity

2D RoPE, like its 1D predecessor, is efficient. The rotation is implemented as channel-wise element-wise multiplication by sines and cosines, with $O(d)$ operations per token:

Linear Complexity: The total cost per attention head is $O(N d)$ , scaling linearly with the sequence or patch count and embedding dimension (Zhang et al., 10 Jan 2025, Heo et al., 20 Mar 2024).
Vectorization and GPU Efficiency: The operation is hardware-friendly, being a sequence of SIMD-friendly multiplications and additions.
Backward Computation: Fast linear-time algorithms exist for forward and, under bounded-entry regimes, backward (gradient) passes using polynomial approximation and FFTs (Chen et al., 23 Dec 2024). However, general unbounded cases still admit a quadratic lower bound unless ShETH is violated.

4. Applications: Vision Transformers, Speech, and More

2D RoPE in Vision Transformers

In ViTs, patches at positions $(x, y)$ are encoded via 2D RoPE (Heo et al., 20 Mar 2024, Schenck et al., 4 Feb 2025):

Classification and Detection: For ImageNet-1k, Swin, and COCO, 2D RoPE delivers improved accuracy and, crucially, outperforms absolute or relative bias-based positional encodings in high-resolution extrapolation.
Resolution Extrapolation: Periodic rotation functions generalize seamlessly to larger or previously unseen resolutions, maintaining or even improving detection/segmentation results in out-of-training-distribution settings.

2D RoPE in Speech Recognition

Applied in Conformer architectures for ASR, RoPE replaces relative position encodings (Li et al., 2021, Zhang et al., 10 Jan 2025):

Efficiency: RoPE admits efficient GPU vectorization and reduces both memory and computational overhead versus standard RelPos.
Performance: In LibriSpeech and AISHELL-1, models with 2D RoPE achieve lower error rates and up to 13–21% faster training compared to RelPos.
Streaming/Online Inference: RoPE is compatible with dynamic chunk training required for streaming ASR.

Extensions such as STRING (Schenck et al., 4 Feb 2025), LieRE (Ostmeier et al., 14 Jun 2024), and ComRoPE (Yu et al., 4 Jun 2025) further generalize 2D RoPE:

3D and High-Dimensional RoPE: For robotics, RGB-D images, and videos, coordinate vectors $(x, y, z)$ are mapped to orthogonal matrices via commuting skew-symmetric generator exponentials. This supports exact translational invariance and robust modeling of spatial interactions.
Learned and Contextual Variants: Both trainable rotation matrices and context-dependent frequencies (e.g., CARoPE, (Veisi et al., 30 Jul 2025)) have been proposed to add flexibility and expressiveness.

5. Extrapolation and Resolution Effects

A principal advantage of RoPE, exploited in 2D, is built-in generalization to untrained positions:

Extrapolation Scaling Laws: The base value of the rotary angle's geometric progression ( $\theta_n = \beta^{-2n/d}$ ) governs periodicity and thus extrapolation length (Liu et al., 2023). Both shrinking and expanding the base from its default $10^4$ can yield vastly improved behavior beyond the training context.
Resolution Scaling in Vision: Experiments show up to +2.9% mIoU on ADE-20k and similar boosts in ImageNet accuracy for higher resolutions (Heo et al., 20 Mar 2024, Yu et al., 4 Jun 2025). ID assignments that compress or align token positions (e.g., ID-Align (Li et al., 27 May 2025)) further mitigate long-range decay, improving cross-resolution token interactions.
Long-Term Decay and Resolution: While periodic sines and cosines ensure smooth extrapolation, practical implementations may show decaying correlation (“loss of attention connectivity”) for very distant tokens in large contexts, especially under default parametrizations.

6. Limitations, Generalizations, and Ongoing Research

Several issues and active research directions have been identified:

Dimension Inefficiency: Not all rotary feature pairs are utilized in long-context or long-distance retrieval; high-frequency pairs tend to be less useful due to rapid phase variation, leading to “dimension wastage” in attention heads (Chiang et al., 16 Feb 2025).
Attention Sinks and Rotary Offset Features: Low-frequency rotary features that do not complete a full period over the sequence can give rise to U-shaped dot product contributions, known as “attention sinks” (Jonasson, 3 Mar 2025).
Separability: Standard RoPE’s axis-aligned construction may limit its ability to model complex or diagonal relationships; mixed or learned basis transformations (as enabled in STRING or via maximum Abelian subalgebra expansions) overcome this limitation (Liu et al., 7 Apr 2025, Schenck et al., 4 Feb 2025).
Wavelet Analysis: RoPE can be viewed as a fixed-scale (Haar-like) wavelet transform; richer, multi-scale positional transforms (e.g., Ricker-based wavelets) have been proposed to increase extrapolation and long-range attention coverage (Oka et al., 4 Feb 2025).
Rotary Generalizations: Extensions leveraging matrix Lie group theory enable richer, multidimensional, and learnable rotary encodings (e.g., LieRE (Ostmeier et al., 14 Jun 2024), ComRoPE (Yu et al., 4 Jun 2025), context-aware RoPE (Veisi et al., 30 Jul 2025)).

7. Practical Integration and Implementation Guidance

API and Libraries: 2D RoPE is available in major frameworks (e.g., Huggingface Transformers’ RoFormer for NLP; code for ViT/2D RoPE via (Heo et al., 20 Mar 2024), STRING (Schenck et al., 4 Feb 2025)).
Parameterization Choices: The choice between axial, mixed, or learned rotations depends on downstream task: axial RoPE suffices for many grid-structured vision tasks; mixed or learned-frequency variants are needed for diagonal/cross-feature dependency.
Fine-Tuning Behavior: Modifying the rotary base enables efficient adaptation to new context lengths or resolutions without extensive retraining (Liu et al., 2023).
Efficiency and Overhead: The overhead of 2D RoPE is negligible (∼0.01% of ViT-B FLOPs); backward passes can be approximated in almost-linear time given bounded key values (Chen et al., 23 Dec 2024).

In summary, 2D Rotary Position Embedding provides a mathematically principled, computationally efficient, and highly adaptable framework for encoding multi-dimensional position information in attention-based models. Through explicit rotations in embedding space, it unifies absolute and relative representation, scales robustly to longer contexts and higher input resolutions, and adapts flexibly to domains including vision, speech, and robotics, as substantiated by recent theoretical and empirical research.