DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling (2001.00987v1)

Abstract: We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large dataset containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

• The paper introduces a novel non-parametric sampling approach to generate depth maps from video, overcoming static scene assumptions.
• It employs candidate matching with SIFT flow and global optimization to ensure spatial accuracy and temporal consistency.
• Experimental results on MSR-V3D, NYU Depth, and Make3D benchmarks demonstrate state-of-the-art performance in depth estimation.

Overview of "DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling"

The paper "DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling" introduces a novel method for generating depth maps from videos. The authors, Karsch, Liu, and Kang, present an approach that overcomes limitations in conventional depth estimation techniques, especially in scenarios with non-translating cameras and dynamic scenes.

Methodology

The proposed method employs a non-parametric approach to sample depth from candidate RGBD images or videos, selected based on image feature similarity. This depth transfer technique can manage both static images and video sequences. For videos, temporal consistency is ensured through optical flow, and motion cues are used to enhance the depth estimation of dynamic objects.

The pipeline consists of three primary stages:

1. Candidate Matching and Warping: From a database of RGBD images, a set of candidate images similar to the input are selected and spatially aligned using SIFT flow.
2. Depth Optimization: A continuous global optimization framework is employed to generate a consistent depth map that incorporates data fidelity, spatial smoothness, and a learned prior.
3. Temporal Coherence: Additional terms are integrated to ensure smooth depth transitions over time for video sequences, accounting for moving objects through motion segmentation.

Dataset and Evaluation

The authors introduce the MSR-V3D dataset, a stereoscopic RGBD video dataset used for training and evaluation. The dataset is acquired using dual Kinect cameras, providing significant variability in indoor and outdoor settings. The technique demonstrates superior results on benchmark datasets when compared against existing methods such as Make3D.

Results and Implications

Numerical results from experiments on datasets like Make3D and NYU Depth illustrate that the method not only achieves state-of-the-art performance on multiple error metrics but also exhibits robust generalization across diverse scenes. The method's ability to effectively convert 2D video content into 3D format further emphasizes its potential utility in automated 2D-to-3D video conversion for the growing 3D media industry.

Conclusion

The paper contributes a practical solution to depth extraction challenges in non-ideal video sequences. While conventional methods like structure-from-motion assume static scenes and parallax-based camera motion, this approach broadens the application range, offering new opportunities for real-time 3D video applications. Future developments may explore enhanced integration with machine learning to further refine depth accuracy and tackle more complex scenes and movements.

Overall, the work provides a significant step forward in depth estimation, with potential implications for improved scene understanding, augmented reality applications, and enhanced 3D content creation.