MRoPE-I: Multi-Axis Rotary Positional Encoding

• MRoPE-I is a novel multi-axis rotary positional encoding method that interleaves per-axis frequencies to provide full positional information across text, image, and video modalities.
• It achieves improved positional coherence and full-frequency utilization, yielding performance gains up to +2.59 compared to head-wise and block-wise approaches.
• MRoPE-I integrates with multimodal transformers with minimal overhead, preserving pretrained textual priors while enabling robust joint processing of spatial, temporal, and linguistic cues.

Interleaved Multi-axis Rotary Positional Encoding (MRoPE), specifically the MRoPE-Interleave (MRoPE-I) variant, is a positional encoding method designed for multimodal transformers handling both language and vision data, accommodating 1D token sequences (text), 2D spatial grids (images), and 3D structures (videos). MRoPE-I is motivated by deficiencies in prior multi-axis rotary positional encoding strategies, targeting comprehensive frequency allocation, positional coherence, and the preservation of pretrained textual priors in vision–LLMs. By interleaving rotary frequencies across spatial and temporal axes at the channel level, MRoPE-I enables each transformer head to access full multi-axis positional information, providing a minimal-change yet empirically robust enhancement over axis-wise splitting or head-wise frequency partitioning strategies (Huang et al., 27 Oct 2025).

1. Motivation and Design Principles

MRoPE-I addresses three essential criteria arising in multimodal transformer architectures:

1. Positional Coherence: Locality in the input (e.g., adjacent image patches or frames) should be reflected in smooth, consistent rotary angle transitions, preserving spatial and chronological structure in attention patterns.
2. Full-Frequency Utilization: Each axis (height, width, and if present, time) must access the complete range of rotary frequencies to accurately encode both fine-grained and large-scale positional differences, avoiding axis-specific frequency starvation.
3. Preservation of Textual Priors: The method should degrade to standard 1D RoPE when applied to text, ensuring pretrained LLMs’ position handling remains intact upon vision or multimodal extension.

These principles distinguish MRoPE-I from conventional axis-wise or head-wise RoPE variants.

2. Mathematical Formulation

2.1 Standard Rotary Encoding

For an embedding dimension $d$ (even), standard 1D RoPE defines angular frequencies $\theta_i = 10000^{-2i/d}$ for $i = 0,\ldots,d/2-1$. The 2×2 block diagonal rotation per position $m$ acts as

$$x'_{2i:2i+1} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix} x_{2i:2i+1}$$

2.2 Multi-axis Bijection and the Interleaving Principle

For spatial coordinates $(h,w)$ in an $H \times W$ image, per-axis frequencies $\{\theta^h_i\}$ and $\{\theta^w_i\}$ are defined for height and width respectively. MRoPE-I weaves these frequencies channelwise: for the $p$-th frequency pair,

• If $\theta_i = 10000^{-2i/d}$0 is even: assign height frequency $\theta_i = 10000^{-2i/d}$1
• If $\theta_i = 10000^{-2i/d}$2 is odd: assign width frequency $\theta_i = 10000^{-2i/d}$3

Formally, for $\theta_i = 10000^{-2i/d}$4:

$\theta_i = 10000^{-2i/d}$5

The positional rotation for token at $\theta_i = 10000^{-2i/d}$6 across all channels is:

$\theta_i = 10000^{-2i/d}$7

with $\theta_i = 10000^{-2i/d}$8. This mechanism generalizes to 3D (video): channels are divided among temporal, height, and width axes and interleaved accordingly.

3. Comparison to Alternative Rotary Encoding Schemes

MRoPE-I’s approach is contrasted with block-wise and head-wise rotary frequency allocation:

Scheme	Per-axis frequency coverage	Cross-axis coupling	Implementation overhead
Block-wise split	Half per axis	None	Low
Head-wise split	Full per axis	None per head	Moderate
MRoPE-I	Full per axis	Yes (per head)	Low (extra permutation)

Block-wise RoPE starves each axis of frequency diversity. Head-wise RoPE (MHRoPE) partitions heads by axis, so no head accesses both dimensions simultaneously. MRoPE-I interleaves frequencies so every head can jointly encode all axes, providing richer positional information within each attention context (Huang et al., 27 Oct 2025).

4. Implementation Details

MRoPE-I can be implemented via a simple permutation and application of the axis-interleaved rotary frequencies. For each token coordinate $\theta_i = 10000^{-2i/d}$9, construct the interleaved frequency list $i = 0,\ldots,d/2-1$0 as described. For each channel pair $i = 0,\ldots,d/2-1$1, compute $i = 0,\ldots,d/2-1$2 for $i = 0,\ldots,d/2-1$3, and apply the corresponding 2×2 rotation block to the embedding. This process is repeated for query and key projections prior to attention computation.

No modifications to the transformer’s feedforward or attention mechanisms are necessary. Integration is a direct swap of the RoPE frequency lookup and rotation logic. For models with an embedding dimension $i = 0,\ldots,d/2-1$4, a common quota is $i = 0,\ldots,d/2-1$5 for height and width, and $i = 0,\ldots,d/2-1$6 for time in video tasks, yielding an allocation ratio of $i = 0,\ldots,d/2-1$7 (Huang et al., 27 Oct 2025).

5. Empirical Evaluation

Across a range of multimodal benchmarks—MVBench, STAR, VideoMME, LVBench, MLVU, Charades, and grounding—the following results are reported (Huang et al., 27 Oct 2025):

• Standard RoPE overall: 62.36
• MHRoPE: 64.75 (+2.39)
• MRoPE-I: 64.95 (+2.59)

Ablation studies indicate interleaving is crucial; removing it reduces visual-token attention in deeper layers by approximately 10%. Altering the frequency allocation ratio from 24:20:20 to 32:16:16 results in a decrease from 64.95 to 63.29. MRoPE-I achieves consistent improvements across general and fine-grained multimodal understanding tasks with no additional model parameters and negligible compute overhead.

6. Practical Integration and Recommendations

MRoPE-I is compatible with pretrained weights and preserves the behavior of standard RoPE for text-only modalities. The recommended practice is to precompute per-axis frequencies, weave them as per the derivation above, and apply the rotations in-place. With images of size 512×512 or videos up to 32k-frame lengths, a smaller YaRN scale (approximately 0.75 of that used for vanilla RoPE) yields optimal results. For video, a temporal stride of 1 balances accuracy and resource utilization. No architectural changes to the transformer are required for deployment (Huang et al., 27 Oct 2025).

7. Broader Context and Significance

MRoPE-I generalizes the concept of rotary positional encoding to multi-axial and multimodal data in a manner that is agnostic to the underlying modality architecture, positioning it as a plug-and-play upgrade for existing vision–language transformer stacks. It directly addresses known limitations of axis-aligned rotary schemes and outperforms prior axis- and head-splitting designs in coverage and empirical results, supporting effective joint processing of spatial, temporal, and textual signals (Huang et al., 27 Oct 2025). A plausible implication is improved modeling for tasks with fine spatial, temporal, or cross-modal dependencies.