Extended Rotary Position Embedding (ERoPE)

• The paper introduces ERoPE as a novel extension that incorporates dimension-adaptive, distribution-aware transformations to improve positional encoding in transformers.
• ERoPE is a framework that generalizes traditional rotary embeddings to support multidimensional data, boosting context scaling in language, vision, and time-series tasks.
• The method enhances long-context reasoning by reducing extrapolation errors and enabling robust multimodal fusion through context-aware and dynamic frequency mechanisms.

Extended Rotary Position Embedding (ERoPE) generalizes the rotary position embedding (RoPE) family to address extrapolation, modality adaptation, context scaling, and fusion requirements across language, vision, time-series, and generative modeling pipelines. ERoPE advances the core concept of position-dependent rotations in Transformer architectures by incorporating dimension-adaptive, distribution-aware, context-sensitive, and layout-staggered mechanisms. These design choices are motivated by theoretical analyses, empirical evaluations, and task-specific constraints documented in recent research, particularly for long-context LLMs, video understanding, multimodal fusion, and generative compositing.

1. Mathematical Foundations and Relational Properties

ERoPE builds on the mathematical principle that position encoding via rotation should satisfy relativity and reversibility properties across 1D, 2D, and N-dimensional input domains. Let $x = (x^{(1)},...,x^{(N)})$ be the position vector, and $\{B_1,...,B_N\}$ a set of skew-symmetric generators in a maximal abelian subalgebra (MASA) of $\mathfrak{so}(n)$; then ERoPE defines the positional transformation as:

$$R_x = \exp(x \cdot B) = \exp\left(\sum_{i=1}^N x^{(i)} B_i\right)$$

This construction ensures relativity:

$R_{x_1}^T R_{x_2} = R_{x_2-x_1}$

and reversibility: $R_{x_1} = R_{x_2} \implies x_1 = x_2$ within a suitably chosen frequency range. Standard RoPE is a special case corresponding to the maximal toral subalgebra, with independent $2 \times 2$ rotations per pair of feature dimensions. ERoPE extends this by supporting learnable basis transformations (parameterized e.g. via a matrix $Q$) for controlled inter-dimensional interactions (Liu et al., 7 Apr 2025).

2. Dimension-adaptive and Distribution-aware Scaling

Long-context extension in LLMs and signal domains is constrained by how distributional properties of rotary angles ($m\theta_i \bmod 2\pi$) shift when input lengths are scaled. ERoPE corrects the one-size-fits-all context interpolation by estimating rotary angle distributions per dimension during pretraining and extrapolation. The strategy is to minimize the KL-divergence between distribution $P_L$ (pretrained) and $P_{L'}$ (extended), by choosing on a per-dimension basis between interpolation (frequency scaling) and extrapolation:

$$\hat\theta_i = \begin{cases} \theta_i/s & \text{if } D^i(P_{L'}^\mathbb{E}, P_L) > D^i(P_{L'}^\mathbb{I}, P_L) + t \ \theta_i & \text{otherwise} \end{cases}$$

where $s = L'/L$ and $t$ is a threshold (Wu et al., 2 Oct 2024). This approach achieves up to 72% reduction in distributional disturbance for 8k context extension and preserves short-context accuracy. ERoPE’s adaptive scaling particularly emphasizes the preservation of low-frequency dimensions, shown to dominate long-distance dependency modeling and to maintain attention integrity in retrieval tasks (Hong et al., 11 Oct 2024, Wu et al., 2 Oct 2024).

3. Context-Aware and Dynamic Rotary Frequencies

Traditional RoPE uses static, context-agnostic frequency schedules across all heads and embeddings. Extended variants such as CARoPE introduce dynamic frequency patterns per attention head and token embedding, computed as:

$\varphi_i^{(h)}(m) = \sum_{t=1}^m f(x_t)_h^i$

where $f(x_t) = 1/(\text{softplus}(x_t W) + 1)$ for a learned projection $W$. This injects context sensitivity, allowing the embedding frequencies to respond to semantic, syntactic, and input-specific cues (Veisi et al., 30 Jul 2025). Experimental benchmarks demonstrate that CARoPE delivers significantly lower perplexity and faster throughput compared to static RoPE baselines.

4. Modality Extension and Multidimensional Generalization

To address video compositing, multimodal fusion, and continuous time-series, ERoPE generalizes the rotational mechanism to multidimensional spatial and temporal coordinates. For video, the positional encoding is restructured from 1D to 3D with low-frequency allocations for the temporal axis (critical to suppress oscillatory collisions), symmetry-preserving diagonal layouts, and adjustable temporal scaling $\delta$ to decouple spatial and temporal granularities (Wei et al., 7 Feb 2025). In time-series and image modeling, ERoPE leverages axial decomposition, assigning partitioned RoPE blocks to each independent coordinate for continuous and irregular indexing (Zivanovic et al., 26 May 2025).

For generative video compositing tasks with unaligned foreground/background layouts, ERoPE staggers the positional labels of composite tokens by applying explicit positional offsets $\Delta$:

$$z'_{\text{foreground}} = z \cdot \cos(\theta(p+\Delta)) + z^\perp \cdot \sin(\theta(p+\Delta))$$

This prevents artificial spatial alignment and preserves contextual boundaries during self-attention fusion (Yang et al., 2 Sep 2025).

5. Attention Patterns, Outlier Control, and Multi-resolution Mechanisms

Comprehensive attention analyses reveal that certain rotary features become outliers, forming "attention sinks" due to frequency bands not completing full cycles over context length. These are formally bounded:

• Rotary frequency for offset: $\theta_i < (2\pi)/p_{\mathrm{max}}$
• Initial angle for persistent negative dot product: $\phi_i > \pi + (p_{\mathrm{max}} \theta_i)/2$

Effective ERoPE design reduces the prevalence and dominance of such outlier features by balancing rotational frequencies and constraining their angular ranges. Additionally, ERoPE’s wavelet-like multi-resolution decomposition facilitates scale-invariant attention, enabling models to process both global and local dependencies optimally and to spontaneously develop multi-band interaction patterns, analogous to wavelet transforms (Ruscio et al., 23 Oct 2024). Fourier-based extensions, such as FoPE, further mitigate spectrum leakage by representing each positional embedding as a complete Fourier series and zeroing out undertrained frequency components (Hua et al., 23 Dec 2024).

6. Cross-modal Decoupling and Geometric Encoding

In multimodal vision-language pipelines, ERoPE must ensure that text and image tokens do not inadvertently inherit positional biases from one another. Circle-RoPE demonstrates that remapping image token indices onto circular trajectories orthogonal to the text index direction, forming a cone-like 3D structure, achieves equidistant encoding (measured by Per-Token Distance, PTD) between text and image tokens:

$$\text{PTD} = \frac{1}{N_{\text{image}} N_{\text{text}}} \sum_{t}\sum_{i} | D_\mathrm{abs}(t, i) - \overline{d_t} |$$

with $\overline{d_t}$ as the mean Euclidean distance post-RoPE (Wang et al., 22 May 2025). Additionally, employing staggered/alternating geometry encoding (AGE), where layers switch between M-RoPE and Circle-RoPE, exploits complementary strengths across hierarchical representations.

7. Practical Impact and Future Directions

ERoPE formalizes the extension, adaptation, and refinement of rotary positional encoding for long-context sequence modeling, video and spatial fusion, time-series learning, and multimodal integration. Objectively, these advances yield:

• Substantially reduced context-length extrapolation errors, with lower entropy and preserved attention patterns (Zhong et al., 19 Jun 2024).
• Robust compositionality for generative video editing where composite elements require non-aligned spatial coordination (Yang et al., 2 Sep 2025).
• Improved velocity estimation and dynamic modeling in temporal detection pipelines (Ji et al., 17 Apr 2025).
• Greater versatility in irregular, multidimensional data domains, surpassing specialized architectures (Zivanovic et al., 26 May 2025).
• Increased interpretability and principled design space via the Lie-theoretic blueprint (Liu et al., 7 Apr 2025).

Future research prospects include deeper integration of context-aware frequency adaptation, learning orthogonal transformations for inter-modal interactions, spectrum regularization, and scalable geometric encoding for multimodal fusion. The architecture-level insights from hybrid attention schemes (interleaving RoPE and NoPE) (Yang et al., 30 Jan 2025) further highlight the importance of context-specific pattern retention for ultra-long context reasoning.

ERoPE thus stands as a mathematically rigorous, empirically validated, and modality-general framework for next-generation positional encoding, facilitating stable extrapolation, robust non-linear fusion, and enhanced multimodal information integration in transformer architectures.