ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning

Abstract: Monocular video depth estimation requires temporal consistency, geometric accuracy, and generalization across diverse scenarios, yet existing methods struggle to achieve all three simultaneously. Discriminative models excel at per-frame accuracy but suffer from temporal drift due to limited context windows, while generative methods improve consistency and generalization at the cost of extensive training data (10M+ samples) and lack of geometric precision. In response to these issues, we introduce \textbf{ICDepth}, a framework that adapts pre-trained text-to-video diffusion transformers for video depth estimation via In-Context Conditioning (ICC), leveraging their rich spatial-temporal priors. To address key challenges in transferring ICC from generation to dense prediction, we propose: (1)~\textbf{SAND-Attention}, which ensures precise spatial-temporal alignment via shared RoPE and enforces unidirectional attention to prevent noise contamination; (2)~\textbf{SRFM}, which injects DINOv2 semantic and resolution priors to enhance geometric precision. ICDepth achieves state-of-the-art results on multiple benchmarks with remarkable data efficiency, trained on only 0.8M frames ($6$--$13\times$ less than competing generative methods), while demonstrating strong zero-shot generalization to diverse domains.

• The paper introduces ICDepth, a novel framework that integrates in-context conditioning with diffusion transformers to achieve state-of-the-art video depth estimation.
• It employs SAND-Attention and the SRFM module to enhance spatial-temporal consistency and geometric accuracy across diverse video domains.
• Training on only 0.8M frames, ICDepth achieves significant gains in metrics like AbsRel and δ1, demonstrating efficient and robust performance.

ICDepth: In-Context Conditioned Video Diffusion Transformers for Robust Video Depth Estimation

Background and Motivation

Monocular video depth estimation—inferring dense per-frame depth in temporally coherent form from RGB video—remains a fundamental challenge. Recent progress in single-frame depth regression has failed to translate robustly to videos due to the triad of temporal consistency, geometric accuracy, and cross-domain generalization. Discriminative pipelines (e.g., adapting image backbones with temporal heads) suffer from limited temporal context and exhibit drift in long videos, while recent generative methods based on video diffusion models require massive datasets and still underperform in geometric detail and visual precision. The integration of powerful scene-level priors from text-to-video diffusion transformers demonstrates potential for mitigating these issues, if strong cross-modal conditioning and perceptual fidelity can be attained.

ICDepth Architecture

ICDepth leverages the Wan2.1 text-to-video diffusion transformer backbone, adapting it for dense prediction via an In-Context Conditioning (ICC) paradigm. The core design integrates clean conditioning signals directly at the token sequence level, enabling the pretrained attention mechanism to explicitly model RGB–depth correspondences across long spatial-temporal contexts.

Figure 1: The proposed ICDepth framework with SRFM and SAND-Attention, showing RoPE-aligned spatial-temporal token integration and unidirectional attention design.

In-Context Conditioning

ICC concatenates the RGB and depth latents as separate tokens along the sequence dimension, providing the transformer with direct, unprojected RGB context. This exploits full spatial-temporal self-attention and avoids the implicit, lossy fusions inherent to channel-wise concatenation, realizing strong cross-modal alignment.

SAND-Attention

Standard ICC in diffusion transformers is ill-posed for dense estimation because full bidirectional attention allows noise from corrupted depth tokens to pollute clean RGB features, and RoPE misalignments disrupt positional coherence. ICDepth introduces SAND-Attention, which enforces:

• Shared RoPE positional encodings, ensuring spatial-temporal consistency between paired RGB and depth tokens.
• Decoupled, unidirectional attention: clean RGB tokens are only queried, not updated, by noisy depth queries. Bidirectional information flow is blocked to prevent noise contamination.
• Zeroed timestep embeddings for RGB tokens, removing unnecessary diffusion noise injection into the conditioning path.

This reconfiguration preserves the clean, stable conditioning which is critical for perceptual tasks, while maintaining efficient attention mechanisms compatible with FlashAttention kernels.

Semantic and Resolution Priors: The SRFM Module

Transformers pretrained for generative tasks such as video generation lack explicit geometric and resolution awareness necessary for precise depth estimation. ICDepth overcomes this with the SRFM module, which:

• Extracts deep semantic guidance from DINOv2 features of the RGB input, enhancing the geometric and object-awareness of the network.
• Encodes spatial resolution dynamically, allowing the model to generalize across arbitrary aspect ratios and spatial sizes without performance deterioration.

These priors are injected via learned scale and shift modulation at the per-block latent level (post-MLP), strengthening both recognition and correspondence capabilities throughout the model’s depth.

Training and Data Efficiency

The framework is trained using flow matching, minimizing a loss computed on masked, valid depth regions across time. Notably, ICDepth is trained on just 0.8M video frames—a significant data reduction (6–13× less) compared to previous SOTA generative models, attributable to the dense spatiotemporal priors inherited from the Wan2.1 backbone and the efficacy of SRFM.

Quantitative and Qualitative Results

ICDepth sets SOTA benchmarks across Sintel, KITTI, Bonn, ScanNet, and ScanNet++ for both absolute and relative metrics in zero-shot and standard evaluation. On Sintel, it achieves a 16% improvement in AbsRel and a 10.1% increase in $\delta_1$ over the next-best approach, with equally strong temporal consistency and competitive per-frame depth.

Figure 2: ICDepth generalizes robustly across challenging domains (fog, night, underwater, animation, various resolutions/aspect ratios) in high-resolution video depth estimation.

ICDepth displays major improvements in flicker reduction and robustness in adverse conditions—such as nighttime (where per-frame and temporal stability are critical), as seen qualitatively and via temporal alignment error (TAE) metrics.

Figure 3: Depth and temporal analysis of nighttime video; ICDepth demonstrates markedly lower flicker and improved geometric accuracy versus prior methods.

Cross-domain evaluations (e.g., animation/gaming/underwater) and OOD generalization (simulated low-light) further highlight its domain transfer capability, with minimal $\delta_1$ performance drop compared to other top-performing models.

Figure 4: Temporal consistency illustration: depth profiles extracted along scanlines show ICDepth's spatio-temporal stability outperforms all competitive methods.

Ablation and Efficiency Analysis

Critical components are individually ablated to validate their impact:

• ICC outperforms channel concatenation drastically in both accuracy and temporal consistency.
• Disabling RoPE alignment or SAND's attention decoupling yields catastrophic performance degredation.
• Removing SRFM entirely or stripping either DINOv2 features or resolution embedding impairs performance and generalizability, especially on non-standard aspect ratios.

Despite the architectural complexity, inference efficiency remains competitive for generative models, with only moderate overhead compared to discriminative feed-forward pipelines, and significantly reduced memory footprint compared to comparable diffusion-based approaches. Only 3–5 diffusion steps are needed for high-quality estimation.

Implications, Theoretical and Practical

ICDepth demonstrates that pretrained generative transformers—if architecturally adapted for clean, positionally aligned ICC and injected with strong semantic and resolution priors—can act as data-efficient, temporally stable dense estimators. It provides compelling evidence for cross-task transfer of foundation model priors when explicit conditioning and information path hygiene are enforced.

Practically, ICDepth offers an extensible framework for temporally consistent, high-fidelity video depth for AR/VR, video editing, and robotics, especially in data-scarce or distribution-variant environments. Theoretically, the results will likely catalyze further work in cross-modal in-context conditioning for dense prediction, and inform the design of hybrid discriminative-generative spatial-temporal transformers. Future research might adapt similar approaches for optical flow, segmentation, or cross-modal representation learning, leveraging large-scale pretrained video diffusion models as universal spatiotemporal priors.

Conclusion

ICDepth establishes a new state-of-the-art for video depth estimation, resolving longstanding issues in temporal consistency, data hunger, and robust generalization by means of targeted architecture advancements in ICC, attention discipline, and prior injection. The model’s design and empirical results position it as a benchmark for future video understanding tasks exploiting foundation model priors under limited supervision.