Interleaved-MRoPE Positional Encoding

• The paper demonstrates that interleaving frequency slots across all axes avoids blind spots and preserves textual inductive priors in multimodal transformers.
• It details a formal construction which interleaves rotary frequencies per head, ensuring full frequency utilization and positional coherence across text and image dimensions.
• Empirical evaluations reveal consistent gains over standard methods in image, video, and referring grounding tasks, leading to improved global and local reasoning.

Interleaved-MRoPE (MRoPE-I) is a multimodal positional encoding scheme designed for joint vision–language transformer architectures. It guarantees that each attention head accesses all positional axes (text sequence, image height, image width) and the full spectral range of rotary frequencies, thereby maximizing positional coherence, frequency utilization, and preservation of textual inductive priors. MRoPE-I is a plug-and-play, architecture-invariant alternative to both standard 1D Rotary Positional Embedding (RoPE) and head-partitioned Multi-Head RoPE (MHRoPE), showing consistent empirical gains across multimodal understanding benchmarks (Huang et al., 27 Oct 2025).

1. Motivating Principles

Interleaved-MRoPE is motivated by three critical principles for scalable, effective positional encoding in multimodal transformer models:

1. Positional coherence: All attention heads must cover the same variety of frequencies on each input axis. Without this, attention can develop "blind spots," and heads restricted to only low or high frequencies for a given axis cannot jointly access both long-range and fine-grained dependencies.
2. Full frequency utilization: The rotary embedding spectrum, ranging from very low (long-range) to very high (fine-grained) frequencies ($1/10000^{0\rightarrow1}$), must be assigned across all axes in every head to maintain granularity.
3. Preservation of textual priors: Pretrained LLM components carry strong linguistic inductive bias through their channel allocation and 1D RoPE structure on text tokens. Approximately the same proportion of channels as in standard text RoPE must be allocated to text for unimpeded linguistic performance.

Standard MRoPE splits the rotational spectrum head-wise, allocating heads strictly to text, height, or width axes, which prevents some heads from seeing certain axes or frequencies. MRoPE-I interleaves the frequency slots for text, image height, and image width within every head, enforcing the above principles.

2. Formal Construction

For a transformer with per-head dimensionality $d_k$, set $P = d_k / 2$ complex slots per head. Denote axes $a \in \{t, h, w\}$ as text position, image row, and image column, respectively. The construction is as follows:

• Frequency schedule:

$$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$

• Axis assignment (interleaving):

Choose channel counts $(C_t, C_h, C_w)$ summing to $d_k$, and set $(P_t, P_h, P_w) = (C_t/2, C_h/2, C_w/2)$. Construct an assignment array $A$ of length $P$ containing $d_k$0 entries of 't', $d_k$1 of 'h', $d_k$2 of 'w', arranged in a round-robin or block-interleaved pattern. Each head uses this array, optionally offset for additional mixing.

• Position index vectors:

For $d_k$3 tokens, define $d_k$4, $d_k$5, $d_k$6 as their per-token text, row, and column indices. For text tokens at position $d_k$7, set $d_k$8, $d_k$9; for image tokens at $P = d_k / 2$0, set $P = d_k / 2$1, $P = d_k / 2$2, $P = d_k / 2$3.

• Rotary application:

Given $P = d_k / 2$4, reshape it to $P = d_k / 2$5 by grouping real pairs. For $P = d_k / 2$6, $P = d_k / 2$7:

$P = d_k / 2$8

$P = d_k / 2$9

Reshape $a \in \{t, h, w\}$0 back into $a \in \{t, h, w\}$1 matching the original input structure.

• Real-valued form: For each rotational slot $a \in \{t, h, w\}$2

$a \in \{t, h, w\}$3

3. Integration within Attention Architectures

MRoPE-I integrates seamlessly into any multi-head self-attention block without architectural changes. The standard steps are as follows:

• Project input tokens $a \in \{t, h, w\}$4 to $a \in \{t, h, w\}$5, $a \in \{t, h, w\}$6, $a \in \{t, h, w\}$7 via learned matrices.
• Reshape to $a \in \{t, h, w\}$8, where $a \in \{t, h, w\}$9 is the number of heads.
• For each head, apply the interleaved rotary encoding to $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$0 and $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$1 using the shared or per-head offset assignment array $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$2 and frequency schedule $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$3.
• Perform scaled dot-product attention, combine heads, and output via a linear projection.

The core difference lies in the per-head rotary step: by interleaving axes and frequencies rather than partitioning them, every attention head is exposed to multi-axis, multi-frequency positional information.

4. Implementation Guidelines

Key practical considerations for MRoPE-I include:

• Both the number of heads $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$4 and per-head dimensionality $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$5 must be even.
• Channel ratios $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$6 must sum to $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$7 and should retain the text fraction used by 1D RoPE to maintain textual bias. Empirical results report $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$8 for $$\omega_i = 1 / 10000^{2 i / d_k}, \quad i = 0, \ldots, P-1.$$9 (thus $(C_t, C_h, C_w)$0), i.e., $(C_t, C_h, C_w)$1.
• Interleaving mask $(C_t, C_h, C_w)$2 may be identical for all heads or circularly shifted.
• Positional ID ranges: For images of size $(C_t, C_h, C_w)$3, row in $(C_t, C_h, C_w)$4, column in $(C_t, C_h, C_w)$5, and for text of length $(C_t, C_h, C_w)$6, text index in $(C_t, C_h, C_w)$7.
• Compatible with extrapolation schemes (e.g., NTK-aware, YaRN): full frequency coverage permits a single global scaling factor to extrapolate to larger spatial resolutions. Notably, $(C_t, C_h, C_w)$8¾ the YaRN scaling required for vanilla RoPE suffices to extrapolate MRoPE-I to $(C_t, C_h, C_w)$9 images.
• The implementation requires only a minor index-lookup and loop for the $d_k$0 rotation per head at the rotary stage.

5. Empirical Performance

Extensive benchmark evaluations demonstrate that MRoPE-I consistently surpasses both standard 1D RoPE and MHRoPE on image, video, and referring grounding tasks:

Task/Benchmark	Vanilla RoPE	MHRoPE	MRoPE-I	Gain (over RoPE)
Image (ImageCLEF, VQAv2)	≃65.9	≃66.2	≃66.7	+0.8
Video (MVBench, STAR, VideoMME)	≃51.9	≃52.1	≃52.4	+0.5
Referring grounding (RefCOCO, Flickr30k)	≃75.2	≃75.7	≃75.9	+0.7
Overall multimodal average	≃64.2	≃64.6	≃64.95	+0.75

On fine-grained spatial tasks (e.g., DocVQA), MRoPE-I produces gains of approximately +1.5 points over 1D RoPE. Attention pattern analysis reveals that the interleaving restores visual focus in deeper layers, aligning with intended modeling capacity (Huang et al., 27 Oct 2025).

6. Context and Applications

MRoPE-I is model-agnostic and functions as a direct substitute for the rotary positional step in any transformer-based vision–LLM. It requires no changes to the model's high-level architecture or attention block design. Empirical findings demonstrate its superiority as a "default" positional encoding for multimodal transformers, providing improved alignment between modalities and enhanced ability for both global and local reasoning. The design also retains compatibility with recent frequency-scaling-based extrapolation schemes, suggesting broad applicability for models tackling larger spatial resolutions.

7. Summary of Contributions

MRoPE-I addresses critical shortcomings of axis/partitioned rotary embeddings in multimodal transformers by:

• Enforcing all-head, all-axis, all-frequency coverage within each attention head.
• Maintaining the alignment with pretrained LLM priors.
• Delivering consistent, measurable accuracy gains across image, video, and referring expression tasks.
• Requiring only negligible changes to implementation.
• Retaining compatibility with state-of-the-art extrapolation regimes.

Its introduction marks a systematic improvement in the design and deployment of positional encodings in vision–language transformer models (Huang et al., 27 Oct 2025).