DVD: Deterministic Video Depth Estimation with Generative Priors

Abstract: Existing video depth estimation faces a fundamental trade-off: generative models suffer from stochastic geometric hallucinations and scale drift, while discriminative models demand massive labeled datasets to resolve semantic ambiguities. To break this impasse, we present DVD, the first framework to deterministically adapt pre-trained video diffusion models into single-pass depth regressors. Specifically, DVD features three core designs: (i) repurposing the diffusion timestep as a structural anchor to balance global stability with high-frequency details; (ii) latent manifold rectification (LMR) to mitigate regression-induced over-smoothing, enforcing differential constraints to restore sharp boundaries and coherent motion; and (iii) global affine coherence, an inherent property bounding inter-window divergence, which enables seamless long-video inference without requiring complex temporal alignment. Extensive experiments demonstrate that DVD achieves state-of-the-art zero-shot performance across benchmarks. Furthermore, DVD successfully unlocks the profound geometric priors implicit in video foundation models using 163x less task-specific data than leading baselines. Notably, we fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.

• The paper presents DVD, a framework that deterministically integrates generative priors to achieve robust zero-shot video depth estimation with enhanced temporal and geometric fidelity.
• DVD employs a timestep anchor and latent manifold rectification to mitigate issues like mean collapse and scale drift in long video sequences.
• The method achieves state-of-the-art performance on benchmarks like KITTI and ScanNet with significantly reduced training data requirements.

Deterministic Video Depth Estimation with Generative Priors: An Expert Analysis

Introduction and Motivation

"DVD: Deterministic Video Depth Estimation with Generative Priors" (2603.12250) decisively addresses the dichotomy in video depth estimation: the superior spatio-temporal priors of generative diffusion models at the expense of stochastic geometric hallucinations and scale drift, versus the efficiency and determinism of discriminative ViT regressors plagued by strong annotation dependency and semantic ambiguity. The proposed DVD framework pioneers the deterministic adaptation of pre-trained video diffusion transformers, unifying fidelity, temporal coherence, and data efficiency—unlocking robust zero-shot video depth estimates using over two orders of magnitude fewer task-tuned examples than SOTA discriminative counterparts.

A synopsis of these paradigm trade-offs is depicted in (Figure 1).

Figure 1: A comparative analysis demonstrates the trade-offs between generative and discriminative paradigms and how DVD resolves these challenges in long in-the-wild sequences.

Core Methodology

DVD's architecture leverages a frozen backbone VAE for latent compression, followed by a deterministic mapping $\Phi: z_x \mapsto z_d$ via a video diffusion transformer (DiT). The method innovates across three technical axes:

1. Timestep as Structural Anchor: Unlike standard diffusion inference, DVD binds the conditioning timestep $\tau$ to a mid-range value, exploiting the pre-trained backbone's spectral bias. This anchor enforces a controlled regime where the network balances global geometric stability with high-frequency boundary detail. Ablation reveals that inappropriate choices for $\tau$ collapse fidelity or induce gross over-smoothing. This anchor is visualized in (Figure 2).

   Figure 2: Timestep anchoring empirically reveals a stability–detail trade-off, highlighting the role of $\tau=0.5$ as ideal for video depth inference.

2. Latent Manifold Rectification (LMR): Single-pass regression in latent space typically induces mean collapse, suppressing fine geometric boundaries and yielding temporal flicker. DVD rectifies this by enforcing both spatial (latent gradient) and temporal (latent flow) differential constraints in $\mathcal{L}_\text{video}$, explicitly regularizing for high-frequency structure and coherent motion. The benefit of LMR, over naïve regression and alternative regularizations, is quantitatively and visually corroborated in (Figure 3).

   Figure 3: Ablation analyses show that each DVD design—LMR, deterministic adaptation, and joint training—markedly improves geometric fidelity and structural robustness.

3. Global Affine Coherence: While memory bottlenecks necessitate sliding-window inference, DVD exhibits an inherent affine consistency across overlapping windows in the VAE-decoded space. Simple closed-form least-squares alignment efficiently corrects global scale/shift drift without complex stitching or iterative temporal modules. This property, diagnostic of deterministic latent mapping, is analyzed in detail in (Figure 4).

   Figure 4: DVD’s outputs exhibit strictly affine-well-behaved window transitions, as confirmed by tight post-alignment residuals and linear inter-window correlation.

The complete DVD pipeline, from VAE encoding to long-video sliding inference, is summarized in (Figure 5).

Figure 5: Schematic of DVD, detailing single-pass DiT regression, LMR, and global affine coherent window alignment for extended video inference.

Empirical Results and Analysis

Geometric and Temporal Fidelity: On KITTI, ScanNet, Bonn, and Sintel, DVD establishes new zero-shot SOTA in both AbsRel and $\delta_1$ (e.g., AbsRel = 6.7 on KITTI, 5.5 on ScanNet) and excels in fine-grained boundary metrics. Qualitative evaluation confirms pronounced improvements in spatial sharpness and the elimination of geometric hallucinations versus generative SOTA, and reduced semantic ambiguity compared to discriminative methods.

Figure 6: Qualitative results show DVD's pronounced boundary fidelity in both indoor and outdoor scenarios.

Data Scalability and Efficiency: DVD achieves these results using only 367K training video frames—comprising $<1\%$ of VDA's annotation footprint (60M frames)—demonstrating drastic data efficiency. Inference is competitive with efficient discriminative architectures and vastly outpaces iterative generative sampling pipelines.

Figure 7: DVD's long-video performance, inference speed, and scale robustness versus the annotation and compute requirements of SOTA methods.

Long-Horizon Consistency: DVD maintains scale and structure over thousands of frames, as shown in complex navigation sequences, outperforming generative methods that drift and discriminative baselines that misinterpret ambiguous regions.

Figure 8: DVD preserves sharp global geometry across lengthy sequences, where alignments rectify local scale while maintaining temporal consistency.

Theoretical Implications and Framework Ablation

Comprehensive ablation elucidates the necessity of each component:

• The structural anchor (timestep $\tau$) is a non-negotiable key to unlocking the pre-trained DiT’s global geometric priors.
• LMR uniquely avoids the mean collapse endemic to pointwise regression, outperforming popular alternatives (e.g., edge-aware, multi-scale gradient matching).
• Joint image-video training prevents catastrophic forgetting and maintains optimal single-frame and sequence-level performance.

Framework generalizes robustly to different video DiT backbones and diverse video domains without architectural or operational change.

Limitations and Future Directions

Failure analysis identifies the boundary of the affine alignment assumption: massive scene transitions, egomotion, or extended occlusions may overwhelm coherence, inducing scale drift typical of long-horizon inference without persistent context. Advancing towards adaptive, possibly non-linear latent alignment or enlarged context windows may further mitigate this.

Further, real-time deployment is prevented by the size and complexity of the DiT backbone. Approaches such as architectural distillation and linear-complexity transformers represent promising solutions. Finer spatial resolutions may require moving beyond VAE tokenization to preserve ultra-fine structures.

Conclusion

The DVD framework demonstrates that deterministic adaptation of video generative priors, when combined with specialized architectural and supervisory constructs—structural anchoring, latent manifold rectification, and affine global consistency—enables robust, data-efficient, and high-fidelity video depth estimation. These insights set a new bar for both theoretical framing and practical deployment of geometric reasoning in unconstrained, long-horizon video, and open new research frontiers in the fusion of generative and discriminative paradigms in 3D perception (2603.12250).