MixD-RoPE: Phase-Aligned Mixed Resolution Embeddings

Updated 9 January 2026

MixD-RoPE is a technique that encodes relative positional information in mixed-resolution settings by aligning phase increments across disparate spatial grids.
It addresses phase misalignment of standard RoPE by resampling key positions to match query strides, enabling training-free integration with pretrained Diffusion Transformers.
CRPA, the main instantiation of MixD-RoPE, achieves improved generative fidelity with better quantitative metrics in image and video denoising benchmarks.

Mixed Dimension Rotary Position Embedding (MixD-RoPE) refers to a family of techniques for encoding relative positional information within attention-based architectures where query and key tokens reside on spatial grids with differing resolutions or strides. Originating from the challenge of stabilizing attention in mixed-resolution denoising with Diffusion Transformers (DiTs), MixD-RoPE identifies and addresses core failure modes in standard Rotary Positional Embedding (RoPE) arising from phase misalignment and aliasing when spatial strides diverge. Its principal instantiation, Cross-Resolution Phase-Aligned Attention (CRPA), systematically realigns phase increments across token pairs, enabling training-free compatibility with pretrained models and supporting high-fidelity, mixed-resolution generative workflows in both image and video domains (Wu et al., 24 Nov 2025).

1. Background: RoPE and Positional Encoding in Transformers

Rotary Positional Embedding (RoPE) maps a token's positional index $p$ to a block-diagonal rotation matrix: $\mathcal R(p) = \bigoplus_{i=0}^{d/2-1} R(\omega_i p), \quad R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \ \sin\theta & \cos\theta \end{pmatrix}$ Here, each $\omega_i$ denotes a fixed angular frequency, typically exponentially spaced such as $\omega_i = 10000^{-2i/d}$ . The attention between a query at position $p_q$ and a key at $p_k$ is modulated solely by their relative positional offset: $(\mathcal R(p_q)q)^\top (\mathcal R(p_k)k) = q^\top \mathcal R(\Delta) k, \quad \Delta = p_k - p_q$ Standard RoPE presumes all token positions are sampled on a commensurate grid. This foundational assumption fails under mixed-resolution settings, where queries and keys may be sampled at different spatial strides.

2. Mixed-Resolution Problem: Phase Aliasing and Attention Collapse

In mixed-resolution DiT inference, input data (e.g., image or video) is divided into patches at varying spatial resolutions, resulting in query and key tokens with distinct spatial strides $s_q$ and $s_k$ . Informally, a prevalent workaround is to linearly interpolate positions from one grid onto the other: $\phi_r(p) = a_r + s_r (p - b_r), \quad r \in \{\mathrm{LR}, \mathrm{HR}\}$ However, this remapping injects minute but structurally devastating phase shifts for high-frequency RoPE channels. Specifically, physical offsets with identical magnitude induce discordant phase increments due to different grid spacings. The phase error for frequency component $\omega_i$ accumulates as: $\delta\theta_i = \omega_i \left( \frac{s_k}{s_q} - 1 \right) k$ Given that $\omega_i$ increases exponentially, even negligible stride ratio mismatches produce substantial angular deviations modulo $2\pi$ . This results in "phase aliasing," whereby the learned relative phase selectivity of attention heads is violated, destabilizing the score landscape. Empirically, this manifests as blur, artifacts, or total collapse in generative tasks.

3. Solution: Cross-Resolution Phase-Aligned Attention (CRPA)

MixD-RoPE, instantiated concretely as CRPA, remedies the above aliasing by ensuring that all key positions are "resampled" into the query grid units prior to applying RoPE rotations. For queries at stride $s_q$ and keys at stride $s_k$ , CRPA prescribes: $p_k^{(q)} = \frac{s_q}{s_k} p_k$ This guarantees that RoPE rotation for the key $k$ is computed using the index as if it were sampled natively at the stride of the query. The relative offset in RoPE phase increments is thus always measured in consistent "query units": $\Delta^{(q)} = \frac{s_k}{s_q}p_k - p_q$ This enforcement of "one attention, one scale" completely eliminates stride-induced phase misalignment.

In practice, the procedure can be integrated into standard attention mechanisms by only adjusting how positional indices are provided to the rotation kernel. All other attention components—multi-head splits, projections, dropout, and so on—remain unchanged.

4. Algorithmic Implementation and Computational Properties

CRPA is characterized by its minimalistic intervention:

Adapts positional indices by scaling key positions by the stride ratio relative to the queries.
Leaves all model weights unchanged.
Applies RoPE rotation using these rescaled indices for keys and the native indices for queries.

Pseudocode for its integration into a standard attention module is as follows:

function CRPA_Attention(Q, K, V, pos_q, pos_k, s_q, s_k):
    alpha = s_q / s_k
    pos_k_q_units = alpha * pos_k
    Q_rot = RoPE_Rotate(Q, pos_q)
    K_rot = RoPE_Rotate(K, pos_k_q_units)
    scores = (Q_rot @ K_rot.T) / sqrt(d_k)
    A = softmax(scores)
    return A @ V

CRPA introduces no additional matrix multiplies compared to standard RoPE-based attention, as the only change is the scaling of positional index inputs. This design ensures zero additional training cost and seamless integration with optimized attention implementations (Wu et al., 24 Nov 2025).

5. Zero-Shot and Training-Free Compatibility

A core property of MixD-RoPE (CRPA) is that it does not require any fine-tuning of pretrained weights. Since RoPE rotation matrices are parameter-free and the only modification involves how indices are provided at inference (or training), existing DiT models can be immediately stabilized for mixed-resolution workloads. When models are fed uniform single-scale inputs, behavior exactly matches the original, preserving performance and convergence properties.

Additionally, CRPA's trivial implementation allows direct deployment in existing frameworks by overriding only the construction of the positional index tensor within the attention kernel (Wu et al., 24 Nov 2025).

6. Empirical Performance and Benchmark Results

CRPA demonstrates substantial quality improvements over previous mixed-resolution positional embedding schemes in both image and video denoising Transformer tasks:

Video (Wan2.1-1.3B on VBench/DOVER): Baseline interpolation variants (PI-LR/PI-HR) yield DOVER scores ≈63.4/35.0 and VBench total 0.717/0.661 at roughly 43 seconds. CRPA attains DOVER 75.34 (vs. 79.12 for full-HR), VBench total 0.770 (vs. 0.766 full-HR) in the same runtime.
Image (Flux.1-dev on MSCOCO): Interpolation methods yield FID 41.45/49.84 and CLIP-IQA 0.425/0.295. CRPA achieves FID 32.04 (best), CLIP-IQA 0.563, MUSIQ 68.88, CLIP score 31.18 (closely matching the full-HR baseline: 31.50 FID / 0.640 CLIP-IQA).
Three-Stage Pipelines: In coarse→mixed→fine denoising, CRPA outperforms the RALU baseline across both 18-step (32.45 vs. 32.91 FID) and 9-step (32.08 vs. 40.49 FID) regimes.
Quality–Cost Tradeoff: CRPA yields rapid improvements in output quality with small fractions of high-resolution tokens, underscoring its efficiency.

Competing approaches such as NTK-aware and YaRN remain strongly suboptimal under mixed-resolution settings. The results affirm that phase-consistent positional encoding is essential for the stability and fidelity of DiT-based mixed-resolution generation (Wu et al., 24 Nov 2025).

7. Relationship to Other Mixed-Dimension Rotary Embedding Schemes

MixD-RoPE, as instantiated by CRPA, addresses a distinct but complementary failure mode to higher-dimensional generalizations such as GeoPE (Yao et al., 4 Dec 2025) and LieRE (Ostmeier et al., 2024). While the latter extend the RoPE framework to multi-axis (SO(3), general SO(n)) settings to encode geometric structure and enhance expressivity for 2D/3D data, MixD-RoPE specifically targets the issue of cross-resolution phase misalignment that arises when attention operates over nonuniform sampling grids. The central insight of "one attention, one scale" in MixD-RoPE is orthogonal to the Lie group generalizations, and no claim is made regarding gains in geometric expressivity. However, both lines of research validate the fundamental importance of precise, physically grounded phase alignment in transformer-based models for vision and generative modeling.

Table: Comparison of Mixed-Dimension RoPE Mechanisms

Method	Target Issue	Key Mathematical Fix
MixD-RoPE/CRPA	Mixed-resolution phase misalignment	Resample all keys into query's stride
GeoPE	Spatial manifold fidelity (2D/3D)	Quaternion and Lie algebra averaging
LieRE	General dimensionality & expressivity	SO(n) Lie group parameterization

Conclusion

Mixed Dimension Rotary Position Embedding (MixD-RoPE), concretely realized as Cross-Resolution Phase-Aligned Attention (CRPA), resolves structural mismatches in rotary embeddings across grids of different resolutions by enforcing consistent phase increments in the attention mechanism. This approach restores the precise relative phase selectivity critical for the stability and generative fidelity of DiTs operating at mixed resolution. CRPA is a training-free, drop-in solution compatible with pretrained models and supersedes earlier mixed-resolution solutions in both efficiency and output quality (Wu et al., 24 Nov 2025).