RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation

Abstract: Modern video diffusion transformers position their tokens through RoPE on the (u,v,t) axes -- a description of the camera's sampling grid that says nothing about the 3D structure of the scene. We observe that the geometric relation between two camera rays is captured by the Plucker reciprocal product, which is bilinear in the two rays -- the same algebraic form as the dot product in Transformer attention. Building on this analogy, we propose RayPE, a positional-encoding extension that injects per-token 6D Plucker coordinates additively into the queries and keys of self-attention, with a query/key flip arrangement under which the symmetric identity configuration coincides exactly with the reciprocal product. The injection is additive, the resulting attention score decomposes into a content term, a geometry term, and two content and geometry cross-terms -- all of which our experiments find individually necessary. To make the encoding stable across video data with heterogeneous camera-translation scales (SfM, deep SLAM, metric), we further decouple ray direction from moment magnitude, gate the encoding by a learned function of the log-magnitude, and apply RMSNorm to align it with the QKNorm-normalized content branch. The full module adds less than 0.1% parameters to a pretrained video DiT, is zero-initialized to start from the pretrained weights, and improves camera controllability, cross-frame 3D consistency, and overall video quality on a four-dataset training mixture.

• The paper introduces a novel additive injection of 6D Plücker coordinates into self-attention, enabling explicit 3D correspondence and improved camera trajectory fidelity.
• It presents a Normalize-Gate-Inject pipeline that decouples scale and stabilizes geometric modulation across heterogeneous datasets while preserving pretrained model structures.
• Empirical evaluations show significant improvements in metrics (FVD, rotational and translational errors) over state-of-the-art camera-conditioned baselines.

RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation

Motivation and Problem Framing

Conventional video diffusion transformers predominantly utilize rotary positional encoding (RoPE) over the $(u,v,t)$ axes, encoding only the sampling grid index and neglecting actual 3D scene structure. This results in an inability to exploit geometric ray correspondences, particularly when synthesizing frames under novel camera trajectories. Camera-aware conditioning in prior works is typically handled via auxiliary encoders, adapters, or cross-attention—processing geometry outside of self-attention, generating attention scores based solely on content similarity and failing to capture explicit 3D relationships. Recent multiplicative camera-aware positional encoding proposals attempt to embed geometric signals into RoPE, but generally disrupt pretrained structure and lack expressive coupling between content and geometry.

Plücker-Ray Attention Encoding

The core advancement in this work is the direct, additive injection of per-token 6D Plücker coordinates (direction and moment) into self-attention queries and keys.

Given two tokens, their associated camera rays are parameterized as $(d, m)$ with $d$ as direction and $m = o \times d$ (ray moment). Plücker reciprocal product—a bilinear SE(3)-invariant measure—vanishes if rays are coplanar, i.e., observe the same 3D location. Self-attention forms $\langle q_i, k_j \rangle$ are bilinear, matching the algebraic structure of the reciprocal product.

RayPE's construction injects Plücker coordinates into queries/keys with a flip arrangement: queries receive $(d,m)$, keys receive $(m,d)$, creating an attention score comprising four interpretable terms—content, geometry, content$\leftrightarrow$geometry cross-terms, and the pure geometry term (the Klein form). All are empirically required for robust 3D consistency and trajectory control.

Figure 1: Illustrates why attention must integrate ray geometry by depicting two cameras each casting rays and factorizing attention as content and ray-geometric similarity.

Normalize-Gate-Inject: Scale-Stable Encoding

Raw Plücker moments scale linearly with translation magnitude, causing instability across datasets (SfM, deep SLAM, metric) with heterogeneous pose conventions. RayPE's NGI pipeline decouples direction from log-magnitude and gates the geometric injection via a learned function $G_s(s)$ based on log-magnitude, then applies RMSNorm for alignment with the QKNorm-normalized content branch.

This ensures robust and adaptive geometric modulation, preserving absolute scale information while remaining invariant to rescaling. Zero initialization maintains compatibility with pretrained weights, and the additional parameters constitute less than $0.1\%$ of DiT-5B.

Figure 2: Shows RayPE integration into self-attention, illustrating the separation between the content pipeline and the geometry pipeline (NGI).

Empirical Evaluation

Quantitative

RayPE is evaluated on Wan2.2-TI2V-5B and Wan2.2-I2V-14B backbones, trained on a mixture of four datasets: RealEstate10K, DL3DV, PanShot, and OmniWorld. Metrics include CLIP similarity, Fréchet Video Distance (FVD), Fréchet Inception Distance (FID), and trajectory errors (RotErr, TransErr, CamMC, ATE) computed using ViPE estimated poses.

RayPE outperforms state-of-the-art camera-conditioned baselines—CameraCtrl, ReCamMaster, UCPE, ReRoPE—across all metrics. Pose errors are reduced to $(d, m)$0 (RotErr) and $(d, m)$1 (TransErr) on DiT-5B, with FVD improved to $(d, m)$2 and CLIP to $(d, m)$3. Results persist under both raw and rescaled evaluation protocols.

Qualitative

RayPE demonstrates superior trajectory following and cross-frame 3D consistency across diverse scenes, including in-domain and out-of-distribution tests. For stylized or cinematic first frames, RayPE reliably executes prescribed camera motion while maintaining artistic characteristics and spatial layout, unlike competing methods which frequently drift from trajectories or collapse motion cues.

Figure 3: Comparison on RealEstate10K showing RayPE's fidelity to target camera motion versus baseline methods.

Figure 4: RayPE gallery on assorted scenes, each row showing a distinct camera trajectory.

Figure 5: Out-of-distribution comparison on artistic paintings; RayPE faithfully follows camera paths for stylized content.

Figure 6: RayPE generalizes to hand-painted concept-art first frames, executing accurate camera trajectories.

Figure 7: Multi-trajectory gallery for fixed movie-still first frames; RayPE preserves content and prescribed motion.

Component and Design Ablations

Ablations demonstrate the necessity of each RayPE component:

• Removing Q/K flip yields marginal degradation, confirming it as a basis choice.
• Omitting NGI (normalization, gating, RMSNorm) substantially worsens trajectory error and FVD.
• Suppressing content$(d, m)$4geometry cross-terms or pure geometry-geometry terms generates notable performance drops, highlighting the essential coupling provided by the architecture.
• Alternative designs—multiplicative RoPE splits with reduced parameterizations—are universally outperformed by RayPE's additive 6D Plücker injection.

Data composition analyses show cross-domain training becomes increasingly beneficial with RayPE, reflecting robustness against pose-scale heterogeneity due to NGI.

Implications and Future Directions

RayPE establishes a canonical framework for embedding explicit 3D ray-space geometry inside the attention dot product in video diffusion transformers, rather than relying on auxiliary content branches or multiplicative encodings which disrupt pretrained structure. The approach is lightweight, minimally invasive, and generalizes across datasets and content domains.

Practical implications include improved camera trajectory control for generative video models and increased fidelity of 3D consistency in synthesized sequences. Theoretical implications relate to the algebraic integration of geometric priors directly into attention computation—potentially extensible to other modalities (e.g., multi-view vision-language transformers, light field rendering).

Potential future directions involve extending RayPE to non-video generative tasks (image-to-3D, single-view novel view synthesis), leveraging the generalized Klein-form geometry for other invariant tasks, and exploring further cross-term couplings in attention for higher-order spatial or physical signals.

Conclusion

RayPE directly and additively injects per-token ray-space geometry into the self-attention mechanism of video diffusion transformers, exploiting the bilinear algebra of the Plücker reciprocal product to encode geometric correspondences. Through NGI, it achieves robust cross-dataset and cross-domain generalization. RayPE achieves superior controllability and cross-frame consistency while preserving baseline generative quality, and represents an explicit, theoretically-grounded approach to 3D-aware generative modeling.