RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video

Abstract: Self-supervised novel view synthesis (NVS) remains challenging to scale, despite the abundance of video data, largely due to the brittleness of training on realistic videos and the hard-to-predict scaling behavior of multi-network system designs. We introduce RayDer, a unified, feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone, turning self-supervised NVS into a well-posed single-model scaling problem. A minimal dynamic state, treated as a nuisance factor, absorbs time-varying content and enables stable training on unconstrained real-world video. Importantly, RayDer keeps static-scene NVS as its target task: dynamic content is leveraged purely as scalable supervision, not reconstructed as in dynamic-scene (4D) NVS. Across multiple model sizes and orders of magnitude in data, RayDer exhibits clean power-law scaling with data and compute, and outperforms static-scene data mixtures. On a large number of benchmarks, RayDer achieves strong zero-shot open-set performance competitive with state-of-the-art supervised approaches. Project Page: https://compvis.github.io/rayder

• The paper presents a unified transformer architecture that integrates camera estimation, scene reconstruction, and rendering into one scalable model.
• It introduces minimal dynamic state variables as nuisance parameters to enable stable and robust training on unconstrained, dynamic video data.
• Empirical scaling analysis shows marked improvements in pose estimation and synthesis quality, positioning RayDer as a strong foundation for 3D vision tasks.

Scalable Self-Supervised Novel View Synthesis via RayDer

Motivation and Problem Statement

Self-supervised novel view synthesis (NVS)—generating unseen views of a scene given input images—is fundamentally limited by the scarcity of curated, static, multi-view datasets. Existing self-supervised NVS methods typically require either static scenes or explicitly pose-annotated data, severely restricting scalability despite the vast availability of real-world videos. Moreover, current system designs segment camera estimation, scene reconstruction, and rendering into separate models, which complicates scaling due to unpredictable interactions and optimization fragility. Dynamic content in videos—including object motion and scene changes—further complicates self-supervised training by entangling motion with pose estimation, often resulting in catastrophic training instabilities.

RayDer addresses these bottlenecks through (i) a unified transformer architecture that jointly learns camera, scene, and rendering tasks, and (ii) explicit modeling of scene dynamics via nuisance variables, enabling robust training on unconstrained, dynamic video data. The objective is to convert self-supervised NVS into a canonical single-model scaling problem, so that learning behavior cleanly tracks data and compute increases, mirroring trends observed in large language and vision models.

Figure 1: Overview of existing approaches and RayDer's capability to leverage unconstrained video containing dynamic objects for NVS, eliminating the static-scene data bottleneck.

Methodology

Unified Transformer Backbone

RayDer consolidates camera estimation, scene representation, and rendering into a single, scalable transformer backbone. This joint approach replaces traditional multi-network NVS designs—such as RayZer, which comprises distinct ViTs for camera, scene, and rendering—with a feed-forward model where all sub-tasks share capacity and training signal. This architectural unification is critical: at fixed parameter counts, RayDer demonstrates improved pose estimation accuracy and novel view synthesis quality relative to equivalent multi-network baselines.

Figure 2: RayDer's architecture merges camera estimation and NVS within a unified transformer backbone, augmented by local high-resolution layers for efficient, detail-preserving processing.

Dynamic State Modeling and Stability

A key innovation is the integration of a minimal per-view dynamic state variable $s_i$ predicted jointly with camera pose $p_i$. This design allows the model to absorb time-varying content, preventing pose representations from degenerating when exposed to dynamic scenes during training. The dynamic state is introduced as a nuisance variable—present at training but omitted at inference—enabling stable NVS learning from real-world videos without requiring explicit 4D dynamic scene modeling.

Empirically, models without dynamic state modeling collapse or diverge when exposed to dynamic data, while the nuisance-state formulation enables robust learning over orders of magnitude in data scale.

Architectural and Training Strategies

• Autoregressive pose learning: To prevent pose representations from capturing sequence order instead of geometry, RayDer employs autoregression over views with a randomized order, ensuring geometric generalization in both sparse and dense observation regimes.
• Parallel-target attention and high-resolution layers: Token-level attention is factorized such that input tokens attend only within inputs, while target tokens attend to inputs and themselves, supporting efficient parallel target view prediction and scalable high-resolution detail via lightweight local layers.

  Figure 3: Demonstration of RayDer's qualitative improvement with increased data and compute; higher data/model scales consistently yield sharper and more accurate synthesis.

Experimental Analysis

Scaling Laws and Compute-Optimal Training

RayDer is trained and evaluated across multiple model sizes (XS–L), dataset fractions (1%, 10%, 100% of SpatialVid), and compute budgets. Scaling analysis reveals clean power-law relationships between test-set performance (MSE, LPIPS, 1-SSIM) and both dataset size and training compute:

Figure 4: RayDer’s compute-optimal Pareto frontier for NVS performance, exhibiting simple and tight power-law dependencies on dataset size and compute.

Notably, data bottlenecks dominate in low-data regimes (large models overfit small data), while larger models benefit disproportionately from additional data. This trend continues even when static-scene datasets are aggregated, demonstrating that unconstrained dynamic video is essential for further scaling.

Camera Pose Estimation: Transferability and Scaling

The unified architecture and autoregressive pose supervision yield pose representations that scale monotonically with more data and compute, and transfer robustly across domains and scenes—unlike previous works where dataset-specific shortcuts were observed.

Figure 5: Camera pose estimation error (rotation/translation) decreases predictably with both data and model size, for both scene-specific and cross-scene transfer tasks.

Open-Set and Zero-Shot NVS Performance

RayDer exhibits strong open-set generalization, achieving near or state-of-the-art results (PSNR, SSIM) across diverse benchmarks: LLFF, DTU, CO3D, WildRGBD, Mip-NeRF 360, and Tanks&Temples. Crucially, these results are achieved via fully self-supervised training, with RayDer using only its own camera pose predictions for inference (strictly harder setting than supervised baselines). Competing models often rely on either pose supervision, hand-crafted priors, or video diffusion pretraining at much higher compute cost.

Figure 6: Zero-shot novel view synthesis on diverse unseen benchmarks; RayDer consistently achieves sharper and more accurate results compared to prior self-supervised and multi-dataset approaches.

Ablations and Comparative Studies

RayDer outperforms pose-supervised models trained with pseudo-ground-truth (MegaSaM) on large-scale video, and achieves competitive closed-set performance on small static datasets (e.g., DL3DV-10k) without sacrificing its large-scale, unconstrained video capability.

Limitations

• Under-constrained Regions: As with other regression-based NVS objectives, RayDer produces low-frequency, blurry estimates in scene regions not observed by any input view. This limitation is inherent to the supervised loss and could be mitigated with generative or uncertainty-aware decoders.
• Dynamic Content during Inference: Because dynamic state is marginalized at inference (i.e., not estimated/tested), dynamic/moving objects are rendered as temporal averages rather than reconstructed explicitly. Extension to true 4D NVS would require large-scale multi-view, multi-dynamics data with explicit disentanglement mechanisms.
• Out-of-Distribution Failure Modes: Domain shifts—such as evaluation on object-centric, black-background datasets like DTU—can result in failure to predict camera geometry, due to mismatch between train and test distributions.

  Figure 7: Both main failure modes—unobserved content and dynamic scene elements—lead to low-frequency “mean” predictions under the regression objective.

Implications and Future Directions

RayDer’s principal contribution is establishing self-supervised static-scene NVS as a scalable, well-behaved learning problem, once system design addresses architectural unification and training stability on dynamic data. This reframes self-supervised NVS as a candidate foundation model task, analogous to LLMs and large vision backbones, with clear scaling trends and broadening capabilities as data/model size increase.

Immediate practical implications include:

• Closed- and open-domain NVS with no pose supervision: Lowering the annotation cost for industrial-scale 3D modeling, AR/VR, and robotics.
• Transferable 3D geometry representations: Improved geometric pretraining for downstream tasks, without curated pose supervision.
• Guidance for future model scaling: Power-law predictability simplifies resource allocation and model design for continued scale-up.

Anticipated future research includes integration with generative decoders for hallucination and uncertainty handling, extension to explicit 4D-dynamic NVS, multimodal interface integration, and hybridization with partial supervision to bridge static and dynamic scene understanding in unconstrained environments.

Conclusion

RayDer represents a significant step toward truly scalable self-supervised novel view synthesis, demonstrating that architectural unification and nuisance dynamic state modeling are sufficient for stable, predictable learning from unconstrained real-world video. This unlocks classic scaling laws, superior sample and pose generalization, and positions self-supervised NVS as a competitive approach relative to both supervised and diffusion-pretrained methods, without reliance on pose annotations or static-scene curation. The framework aligns self-supervised NVS with established foundation model paradigms and motivates further scale-up and adoption for 3D vision tasks in-the-wild.