Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer (1907.01341v3)

Abstract: The success of monocular depth estimation relies on large and diverse training sets. Due to the challenges associated with acquiring dense ground-truth depth across different environments at scale, a number of datasets with distinct characteristics and biases have emerged. We develop tools that enable mixing multiple datasets during training, even if their annotations are incompatible. In particular, we propose a robust training objective that is invariant to changes in depth range and scale, advocate the use of principled multi-objective learning to combine data from different sources, and highlight the importance of pretraining encoders on auxiliary tasks. Armed with these tools, we experiment with five diverse training datasets, including a new, massive data source: 3D films. To demonstrate the generalization power of our approach we use zero-shot cross-dataset transfer}, i.e. we evaluate on datasets that were not seen during training. The experiments confirm that mixing data from complementary sources greatly improves monocular depth estimation. Our approach clearly outperforms competing methods across diverse datasets, setting a new state of the art for monocular depth estimation. Some results are shown in the supplementary video at https://youtu.be/D46FzVyL9I8

• The paper introduces a robust training objective that mixes datasets to improve zero-shot cross-dataset transfer in monocular depth estimation.
• It employs principled multi-objective learning with scale and shift-invariant loss functions to handle inconsistent depth annotations.
• Experimental results show that high-capacity pretrained encoders significantly enhance model generalization for real-world applications.

Overview of "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer"

Monocular depth estimation is a fundamental task in computer vision, challenging due to the absence of stereo information and the inherent ambiguities in single-image cues. Effective monocular depth estimation models require extensive and diverse training data to generalize across a variety of environments. The paper "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer" by Ranftl et al. addresses this challenge by leveraging a principled approach to mixing different datasets, thus improving the robustness and generalization of monocular depth estimation models.

Key Contributions

1. Robust Training Objective:
   • The paper introduces a robust training objective that is invariant to changes in depth range and scale, enabling the combination of datasets with incompatible annotations.
2. Multi-objective Learning:
   • The authors propose the use of principled multi-objective learning to effectively combine data from multiple sources, which is shown to improve generalization performance.
3. Importance of Encoder Pretraining:
   • Highlighting the significance of pretraining encoders on auxiliary tasks, the paper demonstrates that using high-capacity encoders pretrained on large-scale datasets significantly enhances model performance.
4. Experimental Validation:
   • Through rigorous experimentation involving five diverse datasets and the introduction of a new, large-scale dataset derived from 3D films, the authors provide compelling evidence that mixing data from complementary sources improves monocular depth estimation.

Dataset Mixing and Training Strategies

The authors recognize the limitations of existing monocular depth estimation datasets in terms of scale, diversity, and biases. They propose combining multiple datasets, each with its unique characteristics (e.g., indoor vs. outdoor scenes, dynamic vs. static objects, metric vs. relative depth). However, direct combination of datasets poses challenges due to inconsistencies in scale and baseline among different depth annotations.

To address these issues, the authors introduce:

• Scale and Shift-invariant Loss Functions: These loss functions compute objective functions in disparity space, aligning predictions and ground truth in terms of scale and shift. Three specific variants are proposed: mean-squared error (MSE), mean absolute error (MAE), and a trimmed MAE to handle outliers robustly.
• Multi-objective Optimization: A strategy based on Pareto optimization is employed to optimize across multiple datasets concurrently. This principled approach ensures that the loss cannot be reduced for one dataset without increasing it for another, leading to better-balanced model training.

Experimental Results

The paper conducts comprehensive experiments across six diverse test datasets to validate the robustness of the trained models. Notable experimental setups include:

• Zero-shot Cross-dataset Transfer: Evaluating models on unseen datasets ensures a more genuine assessment of generalization capabilities compared to traditional holdout validation on single datasets.
• Comparison with State-of-the-art Methods: The proposed methods outperform existing approaches across multiple benchmarks, affirming the efficacy of dataset mixing and robust training objectives.
• Ablation Studies: These studies highlight the significance of robust loss functions and the impact of different encoder architectures on model performance.

Practical and Theoretical Implications

The paper's findings have several implications:

• Practical: Improved robustness and generalization in monocular depth estimation can significantly benefit applications in autonomous driving, robotics, and AR/VR, where models must operate reliably in diverse and dynamic environments.
• Theoretical: The success of multi-objective optimization and robust loss functions underscores the importance of handling dataset biases and inconsistencies in training models for complex visual tasks.

Future Directions

The work opens several avenues for further research:

• Large-scale Diverse Datasets: Continuing to expand the diversity and scale of training datasets will likely lead to further improvements in monocular depth estimation.
• Cross-modal Training: Integrating additional sensory inputs (e.g., LiDAR, thermal imaging) can provide richer information for depth estimation.
• Real-time Performance: Optimizing model architectures and inference mechanisms to achieve real-time performance is crucial for practical deployment.

Conclusion

The paper by Ranftl et al. makes significant strides towards more robust monocular depth estimation by leveraging diverse datasets and advanced training strategies. The combination of principled loss functions and multi-objective optimization leads to models that generalize well across varied environments, setting a new benchmark in the domain. The approach and findings presented in this work pave the way for more reliable and versatile depth estimation solutions in real-world applications.