Video Depth without Video Models (2411.19189v2)

Abstract: Video depth estimation lifts monocular video clips to 3D by inferring dense depth at every frame. Recent advances in single-image depth estimation, brought about by the rise of large foundation models and the use of synthetic training data, have fueled a renewed interest in video depth. However, naively applying a single-image depth estimator to every frame of a video disregards temporal continuity, which not only leads to flickering but may also break when camera motion causes sudden changes in depth range. An obvious and principled solution would be to build on top of video foundation models, but these come with their own limitations; including expensive training and inference, imperfect 3D consistency, and stitching routines for the fixed-length (short) outputs. We take a step back and demonstrate how to turn a single-image latent diffusion model (LDM) into a state-of-the-art video depth estimator. Our model, which we call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator that is derived from a single-image LDM and maps very short video snippets (typically frame triplets) to depth snippets. (ii) a robust, optimization-based registration algorithm that optimally assembles depth snippets sampled at various different frame rates back into a consistent video. RollingDepth is able to efficiently handle long videos with hundreds of frames and delivers more accurate depth videos than both dedicated video depth estimators and high-performing single-frame models. Project page: rollingdepth.github.io.

• The paper introduces RollingDepth, a novel framework that adapts single-image latent diffusion models to perform video depth estimation.
• The method employs snippet-based multi-frame inference with a modified cross-frame self-attention to ensure temporal coherence in depth maps.
• An optimization-based global alignment and optional partial diffusion refinement deliver high performance on benchmarks like PointOdyssey and ScanNet.

Analysis of "Video Depth without Video Models"

The paper "Video Depth without Video Models" by Bingxin Ke and colleagues presents a novel approach to video depth estimation that uses a single-image latent diffusion model (LDM), circumventing the need for dedicated video models. The authors introduce RollingDepth, an innovative framework that extends a single-image depth estimator to achieve state-of-the-art depth estimation from video sequences.

Methods and Contributions

RollingDepth efficiently transforms a monodepth estimator to process video snippets via multi-frame inference. Unlike traditional video depth methods that rely on large and computationally expensive models, RollingDepth proposes a lightweight and effective technique through two core components:

1. Snippet-Based Multi-Frame Depth Estimation: This component is adapted from a single-image model called Marigold. It maps brief video segments (typically triplets of frames) to depth maps while leveraging a modified cross-frame self-attention mechanism. This mechanism captures the temporal coherence lost in frame-by-frame processing of a video.
2. Optimization-Based Global Depth Alignment: After obtaining depth snippets, the paper introduces a robust registration algorithm to combine these snippets into a coherent depth video. This step uses optimization strategies to ensure temporal consistency across the video length, which the authors argue is crucial for handling sudden changes in depth that arise due to camera movements or dynamic scenes.

Furthermore, RollingDepth offers an optional refinement phase using partial diffusion to enhance spatial details. This progressive approach tackles the challenge of temporal consistency and in-frame accuracy without the computational overhead of full-scale video diffusion models.

Numerical Results

The paper supports its claims by benchmarking RollingDepth against existing state-of-the-art methods across several datasets, including PointOdyssey, ScanNet, and DyDToF. Notably, RollingDepth shows superior performance in terms of both absolute relative error (Abs Rel) and $\delta1$ accuracy. For instance, on the PointOdyssey dataset, a marked improvement is visible with RollingDepth achieving an Abs Rel of 9.6, significantly lower than other models like DepthCrafter and ChronoDepth.

These results indicate a robust capability to maintain accuracy and temporal coherence across diverse sequences, including those with dynamic movements and challenging lighting or texture changes.

Implications

Practically, RollingDepth presents an impactful solution for applications in mobile robotics, AR, and media production, where real-time processing and resource constraints are critical. Theoretically, it suggests an efficient pathway to leverage the strengths of single-image models in video tasks, potentially influencing future research directions toward integrating model architecture refinement with multi-frame temporal dynamics.

Speculations on Future Directions

Future research could explore integrating more advanced forms of snippet refinement leveraging advanced optimization routines or combining other modalities like motion vectors for improved robustness. Moreover, further adaptations to handle extreme depth transitions more precisely and efficiently than existing frameworks could broaden its applicability in the autonomous navigation landscape.

Conclusion

The "Video Depth without Video Models" paper offers a valuable shift in perspective from relying on heavyweight video models to developing lightweight, yet precise, alternatives utilizing single-image diffusion frameworks. Its successful implementation demonstrates the potential of snippet-based processes to achieve comparable, if not superior, video depth estimation performance. Future work could further refine these concepts into broader application-specific methods, enhancing the utility of patient and real-time vision systems.