N-Dimensional Rotatory Position Embedding

• N-Dimensional Rotatory Position Embedding is a mathematically principled encoding method that extends rotary mechanisms into arbitrary N-dimensional domains with explicit relative positional guarantees.
• It leverages Lie algebra foundations, block-diagonal rotations, and group-theoretic properties to inject position-dependent rotations directly into Transformer queries and keys.
• ND-RoPE finds applications across vision, video, and trajectory modeling, offering improved extrapolation, translation invariance, and computational efficiency.

N-dimensional Rotatory Position Embedding (ND-RoPE) is a mathematically principled extension of rotary position encoding mechanisms, originally designed for 1D sequences, to arbitrary N-dimensional domains. These embeddings inject position-dependent rotations directly into Transformer queries and keys, enabling the Transformer’s attention mechanism to capture relative positional or geometric relations with strong extrapolation, computational efficiency, and explicit mathematical guarantees. ND-RoPE unifies a spectrum of approaches—block-diagonal rotations, quaternion averaging, Lie algebra exponentiation, and input-dependent phase selection—anchoring them in Lie-theoretic and group-theoretic foundations. This article surveys the algebraic theory, constructions, empirical properties, implementation patterns, and recent research exemplars.

1. Mathematical Foundations and Theoretical Guarantees

At the foundation of ND-RoPE is the requirement that the positional encoding matrix $R(\mathbf{x}) \in SO(d)$ assigned to position $\mathbf{x} \in \mathbb{R}^N$ satisfies two key properties (Liu et al., 7 Apr 2025, Yu et al., 4 Jun 2025):

• Relativity: The inner product after applying ND-RoPE to queries and keys depends only on the relative displacement:

$$(R_{\mathbf{x}_1} q)^\top (R_{\mathbf{x}_2} k) = q^\top R_{\mathbf{x}_1}^\top R_{\mathbf{x}_2} k = q^\top R_{\mathbf{x}_2 - \mathbf{x}_1} k.$$

• Injectivity/Reversibility: $$R_{\mathbf{x}_1} = R_{\mathbf{x}_2} \implies \mathbf{x}_1 = \mathbf{x}_2$$ within the domain of interest.

The solution set is classified via Abelian subalgebras of the special orthogonal Lie algebra $\mathfrak{so}(d)$. Given generators $\{B_1, ..., B_N\}$ which are linearly independent, skew-symmetric, and pairwise commuting, the embedding is constructed as:

$$R_{\mathbf{x}} = \exp \left( \sum_{i=1}^N x^{(i)} B_i \right), \quad B_i \in \mathfrak{so}(d),~[B_i, B_j]=0.$$

This ensures exact relativity and geometric (periodic) behavior, as shown in comprehensive theoretical analyses (Liu et al., 7 Apr 2025, Yu et al., 4 Jun 2025). The dimension $d$ must satisfy $d \geq 2N$, as MASAs (maximal abelian subalgebras) in $\mathfrak{so}(d)$ have dimension $\lfloor d/2 \rfloor$.

2. Canonical and Enhanced ND-RoPE Constructions

The axis-aligned form arises by block-diagonalizing $R_{\mathbf{x}}$ into $N$ independent $2 \times 2$ rotation blocks, each encoding a scalar linear form in one of the $N$ coordinates (Heo et al., 20 Mar 2024, Su et al., 2021). More generally, frequencies or angle matrices can be learned per axis or block (Yu et al., 4 Jun 2025, Ostmeier et al., 14 Jun 2024):

• Block-diagonal, axis-aligned: Each $B_i$ activates only a $2\times2$ subspace per spatial axis.
• Mixed or generalized: $B_i$ can be any commuting family, and an orthogonal basis change (matrix $Q$) induces cross-axis mixing while retaining the rotary invariance (Liu et al., 7 Apr 2025).

ComRoPE parameterizes the $B_i$ via two sufficient schemes for guaranteed commutativity: Axial-Partition (each block specializes to one axis) and Linearly-Dependent (all blocks are scalar multiples of a base skew-symmetric matrix), which permits scaling to higher N and model widths with robust translation invariance (Yu et al., 4 Jun 2025). GeoPE offers an alternative geometric construction via symmetric averaging in the so($N$+1) algebra (log-exp in the Lie group), ensuring permutation-invariant geometric mean rotations, especially for 2D/3D spatial manifolds (Yao et al., 4 Dec 2025).

LieRE extends this further by allowing $A:\mathbb{R}^N \rightarrow \mathfrak{so}(d)$ as a learnable linear map, unconstrained to axis-aligned cases (beyond block-circulant $B_i$), exploiting the full representational capacity of SO($d$) subject to computation (Ostmeier et al., 14 Jun 2024).

3. Implementation Procedures and Complexity Considerations

The widespread practical implementation leverages the block-diagonal property: the $d$-dimensional head embedding is organized into $d/2$ (or $d/b$ for block size $b$) subgroups, each paired with an angle that is a linear function of the $N$-dimensional coordinate. Each $2 \times 2$ or $b \times b$ block executes a rotation via efficient small-matrix exponentials.

The core steps are:

1. Compute angles for each block from position and (possibly learned) frequency vectors.
2. For query or key vector $x$, split into blocks and apply corresponding rotation matrices: $x'_j = \exp(\sum_{i=1}^N x^{(i)} B_{i,j})\,x_j$.
3. Concatenate rotated blocks and proceed with standard attention calculation.

For advanced variants (GeoPE, LieRE), blocks may be quaternions (SO(3)), or full-skew matrices exponentiated with batched GPU kernels. The computational overhead scales as $O(n d b^2)$ (matrix exponentiation per token per block, with $b$ typically 2, 4, or 8) (Ostmeier et al., 14 Jun 2024, Yao et al., 4 Dec 2025, Yu et al., 4 Jun 2025).

4. Extensions: Input-Dependence, Symmetry, and Nonscalar Rotations

Recent advances generalize ND-RoPE to admit input-dependent angles ("Selective RoPE") (Movahedi et al., 21 Nov 2025). Here, rotation parameters are functions of the query, position, or embedding, introducing dynamic phase control and adaptive relative encoding:

• Angles are produced by neural projections of inputs rather than by fixed increments.
• Nonscalar N-dimensional rotations in SO($N$) can be parameterized by exponentials of learned skew-symmetric generators or as products of Householder reflections.
• Selective RoPE efficiently applies input-dependent block-diagonal rotations, scaling favorably with embedding and sequence length, with performance parity or superiority to fixed-angle baselines in challenging sequence modeling tasks.

GeoPE and similar methods resolve commutativity vs. noncommutativity by averaging logarithms in the Lie algebra before exponentiation, achieving symmetric, permutation-invariant coupling of axes—a critical factor when the geometric structure precludes simple independence (Yao et al., 4 Dec 2025).

5. Applications Across Modalities

ND-RoPE methods are deployed in vision (ViT, 2D/3D images, point clouds), video–text LLMs, structured scene graphs, spatiotemporal foundation models, and multi-agent trajectory generation:

• Vision Transformer: Conventional 2D or mixed-axis RoPE directly boosts image classification, detection, and segmentation, achieving demonstrable accuracy and extrapolation advantages (Heo et al., 20 Mar 2024, Yu et al., 4 Jun 2025).
• Video Representation: VRoPE extends RoPE using symmetric (±) scalar duplications across spatial axes and temporal offsets, seamlessly unifying spatiotemporal and text tokens for Video-LLMs (Liu et al., 17 Feb 2025). This balanced strategy reduces positional attention bias and preserves smooth cross-modal transitions.
• Agent Trajectory Modeling: DRoPE simultaneously encodes relative position and relative heading (angular information), supporting full-graph attention with minimal memory by exploiting rotary periodicity and efficient O($N$) scaling (Zhao et al., 19 Mar 2025).

6. Empirical Properties, Trade-offs, and Limitations

ND-RoPE and its generalizations exhibit consistent performance gains across domains relative to additive or classical relative positional encodings (Yu et al., 4 Jun 2025). Key observations include:

• Memory and compute scale linearly with sequence/object count for block-diagonal construction; quadratic overhead is entirely avoided versus classical RPE.
• Precise commutativity among angle matrices is both necessary and sufficient for translation invariance and relative encoding guarantees (Liu et al., 7 Apr 2025, Yu et al., 4 Jun 2025).
• Empirical ablations confirm superior extrapolation, higher accuracy at increased resolution, and more robust generalization in low-data and high-dimensional settings (Ostmeier et al., 14 Jun 2024, Yu et al., 4 Jun 2025).
• Practical trade-off: increasing block size in rotations yields higher expressive power, at increased computational and parameter cost. Sharing frequencies or tying angle matrices offers additional parameter efficiency.

Limitations include limited support for true rotational entanglement beyond independent 2D subspaces, as full SO($N$) rotations entail noncommuting generators (impractical for large $N$) (Zhao et al., 19 Mar 2025, Yao et al., 4 Dec 2025). Most ND-RoPE schemes do not encode full rigid motions (translations, reflections) or nonorthogonal structure, and performance may plateau for massive head dimensions (Movahedi et al., 21 Nov 2025).

7. Unified Perspective and Future Directions

A systematic Lie-theoretic framework now underpins ND-RoPE design, with maximal abelian subalgebras of $\mathfrak{so}(d)$ providing the solution class. Axis-aligned (block-diagonal), mixed, commutative-learned, and geometric–mean rotations are subsumed in this theory (Liu et al., 7 Apr 2025, Yu et al., 4 Jun 2025, Yao et al., 4 Dec 2025). Input-dependent and context-sensitive rotary phases (Selective RoPE) further enhance flexibility and model capacity (Movahedi et al., 21 Nov 2025).

Prospective directions include:

• Extension to higher-order geometric structures and richer symmetry groups (e.g., SO($N$+1), Clifford algebras, affine or Euclidean groups) (Yao et al., 4 Dec 2025).
• Efficient SO($N$) parameterizations for dense, entangled N-D rotations at scale.
• Integration with learnable context-dependent basis transformations and adaptive Lie algebra generators.

This theoretical and practical unification enables principled positional encoding for any N-dimensional structured data—encompassing not just sequence and image, but arbitrary Euclidean or geometric manifolds within the transformer paradigm.