3D-Factorized Rotary Positional Embedding

• 3D-factorized RoPE is a method that extends traditional rotary embeddings by applying independent rotations along depth, height, and width axes to capture multidimensional spatial relationships.
• It leverages block-diagonal rotation matrices and Lie group extensions to ensure translation-invariance and robust, geometry-aware attention in transformers.
• The approach has advanced applications in vision, robotics, and multimodal tasks by improving accuracy and consistency in spatial, cross-view, and 3D object detection tasks.

3D-factorized Rotary Positional Embedding (RoPE) is a generalization of classic Rotary Positional Embedding for Transformers, designed to encode relative positional and geometric information in high-dimensional structured domains such as images, videos, point clouds, and multi-view scenes. The central principle is to introduce structured, relative position encodings through rotation matrices or their higher-dimensional analogs—factorized to match task geometry (e.g., depth, height, width)—so that the attention mechanisms in Transformers are sensitive to true spatial relations, cross-view consistency, and projective geometry.

1. Fundamentals of 3D-factorized RoPE

Classic RoPE operates over 1D (or at most 2D) coordinate indices by rotating feature subvectors in the complex plane, using angles proportional to position. 3D-factorized RoPE extends this by associating each of several coordinate axes (commonly $d$, $h$, $w$ for depth, height, and width) with a dedicated set of sinusoidal frequency bases and corresponding rotations. For a feature vector $x \in \mathbb{R}^d$ at position $(d,h,w)$ in a 3D grid, the representation is rotated independently or jointly along these three axes, with the transformed features participating in inner-product attention whose outcome depends only on the relative position shifts along all spatial axes.

This coordinate factorization enables explicit, translation-invariant, and geometry-aware positional bias. In its canonical form, the 3D RoPE is constructed as a block-diagonal matrix consisting of $2 \times 2$ rotation blocks per axis, with total rotation being the product (or, in some Lie algebraic extensions, a Lie-averaged or commuted composition) of these axiswise rotations. The attention weight between query at $p$ and key at $q$ is then a structured function of $\Delta_d = d_q - d_p$, $\Delta_h$, and $h$0 (Oztas et al., 25 Jun 2026).

2. Mathematical Construction and SE(3)-Invariant Variants

The most general mathematical formulation of 3D-factorized RoPE can be described as follows:

• For axis $h$1, define a set of frequencies $h$2.
• For token at location $h$3, the per-axis RoPE is

$h$4

• The full 3D-factorized RoPE embedding is the concatenation of $h$5.
• Each $h$6-channel pair of the query/key vector is rotated by the corresponding $h$7 in the complex plane.

Extending beyond axis-wise factorization, several works propose SE(3)-aware variants. URoPE, for instance, achieves cross-view and cross-dimensional invariance by "lifting" 2D RoPE to 3D via depth-anchored ray sampling and projection. Given the 3D position of a key (evaluated along fixed depth anchors), each is projected to the query view and rotated using standard 2D RoPE machinery on the projected coordinates, yielding invariance to global coordinate choice and full compatibility with highly-optimized RoPE kernels (Xie et al., 20 Apr 2026).

In geometric tasks spanning different camera views or 2D–3D correspondences (e.g., novel view synthesis, 3D object detection), this decouples position encoding from any single coordinate chart and restores structured, physically-consistent attention fields.

3. Lie Group and Quaternion Extensions

Non-commutativity is a fundamental challenge in higher-dimensional positional embeddings. Many extensions exploit the Lie group structure of rotations in $h$8 (rotation matrices) or $h$9 (quaternions).

GeoPE addresses noncommuting axis rotations by encoding spatial positions as pure quaternions and constructing a unified, symmetric rotation via Lie-algebra averaging in $w$0: for positions $w$1, each triplet subvector is lifted to a pure quaternion, subjected to base rotations along $w$2 axes, then averaged in the Lie algebra and exponentiated back to $w$3, yielding a commutative, geometrically coupled rotation operator (Yao et al., 4 Dec 2025). The sandwich product $w$4 (for unit quaternion $w$5) rotates the feature triplets, and concatenation reconstructs the positional embedding.

LieRE further generalizes RoPE by learning high-dimensional, block-diagonal skew-symmetric generators, factorizing rotations across axes or channels, and using the exponential map to compute structured, relative-rotational encodings (Ostmeier et al., 2024). These can be optimized per block ($w$6, $w$7, etc.), capturing richer spatial/temporal invariances.

4. Algorithmic Implementation

3D-factorized RoPE is algorithmically efficient. The core steps in practical implementations include:

1. Precompute or learn axis-specific frequency schedules.
2. For each axis, compute sinusoidal positional features for the relevant coordinate.
3. Partition the feature vector into axis-specific channel blocks or channel triplets.
4. Apply the appropriate rotation (complex multiplication, quaternion sandwich, SO(3) block-rotation) to each axiswise or triplet block.
5. Integrate the rotated queries/keys with standard or hybrid attention kernels.

The table below summarizes representative 3D-factorized RoPE variants and their distinctive principles and operations:

Variant	Rotational Structure	Geometric Principle
Standard 3D RoPE	Block-diagonal $w$8 rotations	Factorized by axis
URoPE	2D RoPE after depth-anchored projection	SE(3)-invariant, cross-view
GeoPE	Lie-algebra averaged quaternion rotation	Geometrically symmetric coupling
LieRE	Learned block-diagonal SO(3) exponentials	Lie-group generalization
STRING	Matrix exponentials of commuting generators	Translation-invariant, universal

(Xie et al., 20 Apr 2026, Yao et al., 4 Dec 2025, Ostmeier et al., 2024, Schenck et al., 4 Feb 2025)

5. Applications in Vision, Robotics, Multimodal, and Language Domains

3D-factorized RoPE and its extensions are widely adopted in domains with multidimensional geometric structure:

• Vision transformers and object detection: Accurate restoration of spatial topology and shape bias in ViTs, significantly improving classification, detection, and segmentation benchmarks (e.g., +0.5% ImageNet-1K top-1, +0.2 mAP COCO, +0.9 mIoU S3DIS for GeoPE) (Yao et al., 4 Dec 2025).
• 3D and multiview geometric tasks: URoPE enables cross-view and cross-dimensional geometric reasoning in tasks such as novel view synthesis and depth-aware attention with parameter-free, camera-aware machinery (Xie et al., 20 Apr 2026).
• Video and spatiotemporal transformers: RoPE-3D, VRoPE, and RoPeSLR drive efficient and accurate video modeling, resolving issues of attention bias and sequence discontinuity, and enabling extreme sparsity and sub-linear scaling for long-context synthesis (Liu et al., 17 Feb 2025, Liu et al., 20 May 2026).
• Wireless communications and physical systems: Adaptive 3D-RoPE learns axiswise frequency banks and adapts phase priors for spatio-temporal-frequency modeling in wireless CSI tasks, yielding large NMSE reductions under extrapolation and OOD generalization (Zhang et al., 1 May 2026).
• 3D VLMs and multimodal models: C²RoPE introduces spatio-temporal factorization with spatially-aware causal masking, improving reasoning in large multimodal LLMs (+4.3 EM@1, +18.1 CIDEr on ScanQA) (Ye et al., 11 Feb 2026).
• Diffusion transformers for object motion and relocalization: Depth-aware RoPE for object relocation enables geometry-consistent manipulation and robust scene-level coherence (Oztas et al., 25 Jun 2026).

6. Invariance, Universality, and Theoretical Guarantees

The rigorous mathematical treatment in STRING demonstrates that any smooth, per-token, exactly translation-invariant, matrix-multiplicative position embedding with separability must be representable as

$w$9

where $x \in \mathbb{R}^d$0 are commuting skew-symmetric generators (Schenck et al., 4 Feb 2025). Thus 3D-factorized RoPE is theoretically universal under these invariance criteria, and readily extends to arbitrary coordinate dimension $x \in \mathbb{R}^d$1. Quantum and Lie-theoretic variants harness commutativity and symmetry via canonical structure theorems, restoring topology lost by naive flattening and yielding uniform, geometry-consistent positional fields.

Further, SE(3)-invariance and parameter-free design emerge in designs like URoPE by construction: all transformations depend only on relative intrinsics/extrinsics, and are agnostic to global coordinate system, with no additional learnable parameters beyond standard RoPE frequencies.

7. Empirical Evidence and Impact

Extensive experiments confirm the efficacy and generality of 3D-factorized and geometric RoPE variants:

• Spatial manifold restoration: Improved accuracy, shape bias, and attention structure in vision and robotics tasks (GeoPE, STRING, RoPEMover).
• Cross-view and long-context consistency: Increased loop-closure fidelity, memory persistence, and attention uniformity in world modeling and temporal diffusion transformers (ViewRope, RoPeSLR).
• Physics-aligned extrapolation: Robust zero-shot and out-of-distribution generalization in wireless and spatiotemporal physical tasks (Adaptive 3D-RoPE).
• Translation invariance and universality: Proven theoretical completeness and optimality under translation-invariance constraints (STRING).

These advances have enabled drop-in, efficient, and robust positional inductive biases for a wide range of geometric, multimodal, and sequence modeling applications.

---

Key references: URoPE (Xie et al., 20 Apr 2026), GeoPE (Yao et al., 4 Dec 2025), LieRE (Ostmeier et al., 2024), STRING (Schenck et al., 4 Feb 2025), RoPEMover (Oztas et al., 25 Jun 2026), Adaptive 3D-RoPE (Zhang et al., 1 May 2026), RoPeSLR (Liu et al., 20 May 2026), C²RoPE (Ye et al., 11 Feb 2026), ViewRope (Xiang et al., 8 Feb 2026), VRoPE (Liu et al., 17 Feb 2025), 3D-RPE (Ma et al., 2024).