DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos (2409.02095v2)

Abstract: Estimating video depth in open-world scenarios is challenging due to the diversity of videos in appearance, content motion, camera movement, and length. We present DepthCrafter, an innovative method for generating temporally consistent long depth sequences with intricate details for open-world videos, without requiring any supplementary information such as camera poses or optical flow. The generalization ability to open-world videos is achieved by training the video-to-depth model from a pre-trained image-to-video diffusion model, through our meticulously designed three-stage training strategy. Our training approach enables the model to generate depth sequences with variable lengths at one time, up to 110 frames, and harvest both precise depth details and rich content diversity from realistic and synthetic datasets. We also propose an inference strategy that can process extremely long videos through segment-wise estimation and seamless stitching. Comprehensive evaluations on multiple datasets reveal that DepthCrafter achieves state-of-the-art performance in open-world video depth estimation under zero-shot settings. Furthermore, DepthCrafter facilitates various downstream applications, including depth-based visual effects and conditional video generation.

DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos

In the paper "DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos," the authors tackle the challenging task of estimating video depth sequences in open-world scenarios characterized by diverse visual content, dynamic motion, and varying camera movements. This paper introduces DepthCrafter, a novel method designed to produce temporally consistent depth sequences without relying on supplementary data such as camera poses or optical flow.

Methodology

DepthCrafter employs a video-to-depth model rooted in diffusion processes, leveraging pre-trained image-to-video diffusion models extended through a three-stage training strategy. This model can generate depth sequences with lengths up to 110 frames, accommodating the need for variable-length sequences within the rich and diverse content typical of open-world video data. The training strategy integrates synthetic and realistic datasets to harvest both precise depth detail and content diversity.

The three-stage training strategy marks a critical contribution, enabling DepthCrafter to adapt to long temporal contexts necessary for maintaining temporal consistency and accurately arranging depth distributions across various video lengths. The first stage aligns the model to the video-to-depth task using a realistic dataset, the second refines temporal layers with extended sequences, and the third enhances spatial precision using synthetic data refinement.

In addressing extremely long video sequences, the paper presents an efficient inference strategy that processes video chunks in overlapping segments, calibrated with a noise initialization method ensuring consistent depth scales across adjoining segments. Seamlessness between segments is maintained through a novel interpolation scheme that minimizes temporal discontinuities.

Evaluation and Results

The evaluations demonstrate DepthCrafter’s state-of-the-art performance across several datasets representing indoor, outdoor, static, and dynamic scenes. DepthCrafter achieves a significant improvement in metrics, such as AbsRel and $\delta_1$, across diverse datasets, asserting its high generalization ability. Notably, the effectivity in temporal consistency—avoiding flickering often attributed to single-image depth models directly applied to videos—is evident from the qualitative and quantitative results.

The paper provides comprehensive ablation studies asserting the effectiveness of training stages and denoising steps, reinforcing the notion that the outlined strategies contribute significantly to the model’s success. It also discusses inference speed benchmarks, positioning DepthCrafter favorably against existing high-performing baselines.

Implications and Future Work

DepthCrafter’s ability to provide fine-grained depth estimation across variable-length videos presents several practical applications. It facilitates enhanced depth-based visual effects, conditional video generation, and supports advancements in mixed reality, autonomous driving, and robotics imaging systems. The paper articulates future potential in optimizing computational efficiency and memory consumption, suggesting avenues such as model distillation and quantization could promote practical deployments.

The implications for AI developments are substantial, signifying the convergence of video generation models with precise depth encoding capabilities. Future exploration might extend the integration of multi-modal data or refining the training pipeline with active learning based on user interaction loops to further tailor the depth estimation accuracy in edge cases prevalent in open-world scenarios.

Overall, the paper proposes a robust architectural framework with considerable implications for AI-based video interpretation systems, seeking to harmonize state-of-the-art diffusion techniques with expansive real-world applicability. The model represents a substantial stride toward overcoming the limitations associated with dynamic scene representation and inconsistent depth perception.