Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories

Abstract: Recovering camera parameters from images and rendering scenes from novel viewpoints have long been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task needs what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. We represent each camera as dense ray pixels (raxels) and denoise them jointly with video frames through Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, jointly generating video and camera trajectory from input images, and generating video from input images along a target camera trajectory. Because the model can both predict trajectories from a video and generate views conditioned on its own predictions, we evaluate it through a closed-loop self-consistency test, demonstrating that its forward and inverse predictions agree. Notably, trajectory prediction requires far fewer denoising steps than video generation, even a few denoising steps suffice for self-consistency. We report results on pose estimation and camera-controlled video generation.

• The paper introduces a unified generative framework that leverages raxel images to jointly model video synthesis and camera pose estimation.
• It integrates a decoupled self-cross attention mechanism in a diffusion-based transformer, boosting cycle self-consistency and synthesis quality.
• Ablation studies confirm that specialized raxel encoding is critical, achieving competitive pose accuracy and superior FID/FVD scores on benchmarks.

Learning a Joint Distribution over Videos and Camera Trajectories with Rays as Pixels

Introduction and Motivation

"Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories" (2604.09429) introduces a unified generative framework for modeling both video data and the underlying camera geometry. This work addresses a longstanding fragmentation in computer vision, where camera pose estimation (inverse geometry) and novel view synthesis (forward geometry) are decoupled—each demanding outputs produced by the other for optimal performance, but suffering brittleness and accuracy loss in sparse or ambiguous visual regimes. The proposed model leverages the formulation of “raxel” images, which encode dense per-pixel camera ray information analogously to RGB pixel data, facilitating joint encoding and processing alongside video frames within a diffusion-based backbone.

Methodological Framework

The authors build on large-scale pretrained video diffusion models (VDMs), specifically adapting them for the dual task of video and trajectory generation/inference through two key innovations: (1) the introduction of raxel representations, spatially aligned per-pixel maps encoding camera ray origins and directions, and (2) the integration of a decoupled self-cross attention (DSCA) mechanism within the spatiotemporal transformer backbone. Key procedural aspects include:

• Unified Per-pixel Ray Encoding: Camera extrinsics and intrinsics are projected into the raxel domain, producing images where each channel encodes a vector sum of ray origin and direction, in a manner compatible with standard VAE-based visual tokenizers. Scene coordinate canonicalization ensures that the resulting representations encode only relative, not absolute, frame relationships.
• Joint Denoising and Flow Matching: The model is trained as a flow-matching generative model over the concatenated latent space of video and raxel tokens—de-noising both modalities jointly. Leveraging the physical alignment of raxels and frames allows application of rotary positional embeddings and symmetric attention strategies, fusing trajectory information with visual data and enabling equivalently efficient bidirectional inference.
• Decoupled Self-Cross Attention: The DSCA separates intra-modal (self-attention) and inter-modal (cross-attention) stages. Self-attention is first applied separately to image and raxel latents, propagating information within each modality. Symmetric cross-attention then injects pose priors into video synthesis and allows visual evidence to refine the geometric trajectory estimate.
• Camera Recovery by Procrustes Alignment: To invert from raxel latents back to explicit camera parameters, the decoded raxel images are rigidly aligned via orthogonal Procrustes, with focal length estimation by robust statistics over back-projected points.

Experimental Evaluation

Cycle Self-Consistency and Ablation

Unlike conditional models which learn exclusively $p(\text{video}|\text{pose})$ or $p(\text{pose}|\text{video})$, this approach models $p(\text{video}, \text{pose})$ jointly. Evaluation via cycle self-consistency—round-tripping between pose prediction and pose-conditioned video generation—demonstrates strong alignment between forward and inverse inference, a property unattainable by conditional-only baselines.

Ablation studies validate that raxel encoding is critical: substituting Plücker embeddings for raxels produces dramatic degradation in quality (FID, FVD, geometric errors). Removal of DSCA or cosine similarity loss yields lesser, but still consistent, declines in self-consistency metrics.

Pose Estimation

On benchmarks including RealEstate10K, DL3DV-140, and Tanks & Temples, the model achieves competitive mean relative rotation accuracy, peaking at only two diffusion steps—highlighting the rapid convergence of geometric structure relative to visual synthesis. Although feed-forward methods such as VGGT outperform in absolute pose accuracy, the unified Rays as Pixels approach retains flexibility for generative and inverse inference in a single model.

Camera-Controlled Video Synthesis

On the same benchmarks, the model outperforms prior methods (e.g., MotionCtrl, VD3D, ViewCrafter, Wonderland, Kaleido) in FID and FVD—despite not using explicit 3D representations or auxiliary geometric positional encodings. This indicates superior synthesis quality and temporal consistency on camera-guided video generation tasks, validating the efficacy of the raxel+DSCA design.

Implications and Future Directions

The model blurs the line between perception and geometry in generative modeling, treating non-visual modalities as image-like signals to fully exploit large-scale pretrained visual architectures. This approach may serve as a template for integrating other structured, non-visual signals (e.g., segmentation, physical state trajectories) into generative systems with minimal architectural disruption.

The practical implications are clear for scenarios involving sparse visual data, ambiguous or degenerate views, or forward/inverse rendering cycles, where standalone pose estimators or conditional video synthesis methods fail. The capacity for closed-loop, self-consistent inference is relevant for embodied agents, multiview scene understanding, and applications where robustness to observation sparsity is required.

However, the model’s generalization capacity is currently limited to static scenes under smooth camera motion; extensions to dynamic environments and integration with rich user- or text-conditioned controls present natural avenues for future work.

Conclusion

The "Rays as Pixels" framework (2604.09429) decisively advances joint video-geometry modeling by leveraging raxel images and decoupled attentional fusion within a generative diffusion backbone. The resultant system unifies pose estimation, novel view synthesis, and camera-conditioned video generation into a single paradigm, achieving strong visual and geometric alignment, superior temporal coherence, and robust cycle self-consistency. Such joint modeling establishes a methodological foundation for future research on multi-modal world models, embodied sim2real video generation, and further exploitation of spatially aligned non-visual modality integration in large generative architectures.