Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV (2403.01569v1)

Abstract: Self-supervised learning is the key to unlocking generic computer vision systems. By eliminating the reliance on ground-truth annotations, it allows scaling to much larger data quantities. Unfortunately, self-supervised monocular depth estimation (SS-MDE) has been limited by the absence of diverse training data. Existing datasets have focused exclusively on urban driving in densely populated cities, resulting in models that fail to generalize beyond this domain. To address these limitations, this paper proposes two novel datasets: SlowTV and CribsTV. These are large-scale datasets curated from publicly available YouTube videos, containing a total of 2M training frames. They offer an incredibly diverse set of environments, ranging from snowy forests to coastal roads, luxury mansions and even underwater coral reefs. We leverage these datasets to tackle the challenging task of zero-shot generalization, outperforming every existing SS-MDE approach and even some state-of-the-art supervised methods. The generalization capabilities of our models are further enhanced by a range of components and contributions: 1) learning the camera intrinsics, 2) a stronger augmentation regime targeting aspect ratio changes, 3) support frame randomization, 4) flexible motion estimation, 5) a modern transformer-based architecture. We demonstrate the effectiveness of each component in extensive ablation experiments. To facilitate the development of future research, we make the datasets, code and pretrained models available to the public at https://github.com/jspenmar/slowtv_monodepth.

• The paper introduces two diverse datasets, SlowTV and CribsTV, featuring over 2 million frames to reduce reliance on ground-truth depth annotations.
• It leverages advanced augmentation techniques and a transformer-based backbone to improve model generalization in various settings.
• The approach achieves near state-of-the-art performance in self-supervised monocular depth estimation, offering cost-effective solutions for real-world applications.

Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV for Self-Supervised Monocular Depth Estimation

Introduction

The quest for accurate 3D structure reconstruction of the world from a single image is an enduring challenge in computer vision, crucial for a plethora of applications such as autonomous driving, robotics, camera relocalization, and augmented reality. Unlike traditional depth estimation algorithms, which rely on stereo vision and triangulation, recent advancements show promising results using neural networks to predict depth from a single image – known as monocular depth estimation (MDE). A significant subset of this area, Self-Supervised Monocular Depth Estimation (SS-MDE), leverages the self-supervised learning paradigm to train models without the need for costly ground-truth annotations.

Recent Developments and Challenges

Significant strides have been made in SS-MDE, particularly through leveraging photometric reconstruction losses between adjacent video frames. However, the lack of dataset diversity, focusing predominantly on urban driving scenes, has curtailed the generalization capabilities of SS-MDE models beyond their training domains. Moreover, the convolutional nature of these models often restricts them to specific image sizes, limiting their application diversity.

Proposed Methodology

To tackle these limitations, Spencer et al. propose an innovative approach through the introduction of two novel datasets: SlowTV and CribsTV, totaling over 2 million training frames. These datasets dramatically expand the variety of training environments to include natural outdoor scenes, underwater environments, and indoor settings. This approach is combined with several significant contributions designed to enhance model generalization. These include learning camera intrinsics and implementing a more robust augmentation regime, such as aspect ratio augmentation, RandAugment, and CutOut, alongside utilizing a modern transformer-based architectural backbone.

Key Contributions and Findings

• Novel Datasets: The introduction of SlowTV and CribsTV, featuring over 2 million diverse training frames, marks a substantial leap in dataset diversity for SS-MDE.
• Enhanced Model Generalization: Through strategic interventions in training regimes and architectural choices, the proposed models showcase remarkable generalization capabilities across diverse settings, often outperforming state-of-the-art supervised methods.
• Self-Supervised Learning Par Excellence: By eliminating the dependency on ground-truth depth annotations and leveraging self-supervised learning, the approach underscores the scalability and efficacy of SS-MDE in real-world applications.

Practical Implications and Theoretical Significance

The achievement of achieving near or surpassing state-of-the-art performance in SS-MDE without relying on ground-truth annotations opens new vistas in computer vision. This work could significantly reduce the cost and complexity of deploying depth estimation models for various applications, from enhancing augmented reality experiences to improving the navigational capabilities of autonomous vehicles in unstructured environments.

Looking Ahead

While this paper marks a significant milestone in SS-MDE, challenges such as dynamic object modelling and achieving metric depth from monocular cues remain. Future work focusing on these aspects, including incorporating optical flow constraints for dynamic object handling, could further refine and expand the practical utility of SS-MDE models. Additionally, developing methodologies for metric depth estimation in a self-supervised framework remains a promising avenue for research.

Conclusion

The paper "Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV" represents a significant step forward in self-supervised learning for monocular depth estimation. By leveraging an unprecedented scale and diversity of training data, combined with methodological innovations, the authors significantly advance the state-of-the-art in SS-MDE, showcasing the potential for wide-scale application and further research in this rapidly evolving domain.