DepthFM: Fast Monocular Depth Estimation with Flow Matching (2403.13788v2)

Abstract: Current discriminative depth estimation methods often produce blurry artifacts, while generative approaches suffer from slow sampling due to curvatures in the noise-to-depth transport. Our method addresses these challenges by framing depth estimation as a direct transport between image and depth distributions. We are the first to explore flow matching in this field, and we demonstrate that its interpolation trajectories enhance both training and sampling efficiency while preserving high performance. While generative models typically require extensive training data, we mitigate this dependency by integrating external knowledge from a pre-trained image diffusion model, enabling effective transfer even across differing objectives. To further boost our model performance, we employ synthetic data and utilize image-depth pairs generated by a discriminative model on an in-the-wild image dataset. As a generative model, our model can reliably estimate depth confidence, which provides an additional advantage. Our approach achieves competitive zero-shot performance on standard benchmarks of complex natural scenes while improving sampling efficiency and only requiring minimal synthetic data for training.

• The paper presents a novel DepthFM approach that uses flow matching to achieve fast and accurate monocular depth estimation while addressing diffusion model limitations.
• The method leverages a pre-trained diffusion model and an auxiliary surface normals loss to enhance training efficiency and precision.
• Training on synthetic data with robust geometric constraints enables the model to generalize effectively to real-world scenes.

DepthFM: A Novel Approach for Fast and Efficient Monocular Depth Estimation

Introduction

Monocular depth estimation holds a central role in leveraging 2D images for understanding the 3-dimensional structure of the environment. This capability is crucial for a wide range of applications including augmented reality, autonomous driving, and robotic navigation. Despite the advances in this field, existing techniques either suffer from quality issues, such as blurry artifacts in discriminative models, or computational inefficiency seen in generative methods like those based on diffusion models. Addressing these limitations, we introduce DepthFM, a model that efficiently maps input images to depth maps with high accuracy and low computational cost. Built upon the flow matching concept with strong zero-shot generalization capabilities, DepthFM marks a significant stride in monocular depth estimation by training solely on synthetic data yet demonstrating impressive results on unseen real images.

Method Overview

DepthFM distinguishes itself through several innovative steps, overcoming the shortcomings of prior approaches. At its core, the model employs a flow matching mechanism, which is fundamentally different from the prevalent diffusion model techniques used in monocular depth estimation. Flow matching enables a direct mapping from the input image to the depth map, leveraging straight trajectories through solution space to achieve efficiency and high-quality predictions. A pivotal aspect of DepthFM's design is the use of a pre-trained image diffusion model as a strong prior, which facilitates efficient training on synthetic data and robust generalization capabilities. Additionally, DepthFM includes an auxiliary surface normals loss, enhancing the model's predictive accuracy by imposing geometric constraints on the depth maps. This comprehensive approach allows DepthFM to excel in both synthetic and real-world scenarios, making it a versatile tool for a range of vision tasks.

Key Innovations

• Flow Matching for Efficiency: DepthFM leverages flow matching to circumvent the slow sampling issues of diffusion-based generative models. This strategy enables fast inference and high-quality depth estimation, evidencing the suitability of flow matching for depth and surface normal estimation tasks.
• Transfer from Diffusion Model Prior: The model uniquely benefits from transferring a strong visual prior from a pre-trained diffusion model, facilitating minimal reliance on real-world images during training. This transfer significantly speeds up the training process while ensuring robust generalization across different scenes and datasets.
• Synthetic Data Training with Real-World Generalization: By exclusively training on synthetic data, DepthFM not only ensures a high degree of versatility but also addresses the challenges associated with collecting and annotating large-scale depth datasets. The model's ability to generalize to real-world images without direct training on such data is a testament to its innovative design and the effectiveness of the incorporated techniques.
• Auxiliary Surface Normals Loss: The inclusion of a surface normals loss further refines the depth estimates by aligning them with geometric constraints derived from the 3D structure of scenes. This component plays a crucial role in improving the quantitative performance of depth predictions.

Practical Implications and Future Directions

DepthFM's state-of-the-art performance, backed by its innovative methodology and efficient training regime, opens new avenues for real-world applications that require accurate and fast depth estimation. Its ability to generalize well to diverse scenes underscores the potential for deployment in various domains, from augmented reality applications to autonomous navigation systems. Looking ahead, the principles and techniques underlying DepthFM offer exciting prospects for advancing depth estimation technologies, inviting further exploration into the integration of flow matching models and generative approaches for enhanced visual perception capabilities.

In conclusion, DepthFM represents a significant leap forward in monocular depth estimation, combining the strengths of flow matching models and diffusion-based priors to achieve remarkable efficiency, accuracy, and generalization capabilities. Its development highlights the potential of leveraging synthetic data and innovative model architectures to address complex vision tasks, paving the way for future research and applications in 3D scene understanding.