DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation (2203.14211v1)

Abstract: This paper aims to address the problem of supervised monocular depth estimation. We start with a meticulous pilot study to demonstrate that the long-range correlation is essential for accurate depth estimation. Therefore, we propose to leverage the Transformer to model this global context with an effective attention mechanism. We also adopt an additional convolution branch to preserve the local information as the Transformer lacks the spatial inductive bias in modeling such contents. However, independent branches lead to a shortage of connections between features. To bridge this gap, we design a hierarchical aggregation and heterogeneous interaction module to enhance the Transformer features via element-wise interaction and model the affinity between the Transformer and the CNN features in a set-to-set translation manner. Due to the unbearable memory cost caused by global attention on high-resolution feature maps, we introduce the deformable scheme to reduce the complexity. Extensive experiments on the KITTI, NYU, and SUN RGB-D datasets demonstrate that our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins. Notably, it achieves the most competitive result on the highly competitive KITTI depth estimation benchmark. Our codes and models are available at https://github.com/zhyever/Monocular-Depth-Estimation-Toolbox.

• The paper introduces DepthFormer, which synergizes Transformers and CNNs to overcome limited receptive fields and enhance long-range context for monocular depth estimation.
• Its hybrid architecture features a Swin Transformer branch and a ResNet-based branch, integrated through a hierarchical aggregation and heterogeneous interaction module.
• Experimental results on KITTI, NYU-Depth-v2, and SUN RGB-D benchmarks demonstrate significant improvements over existing methods, underscoring its practical potential.

Overview of DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation

The paper presents DepthFormer, a novel approach for supervised monocular depth estimation that harnesses the complementary strengths of Transformers and Convolutional Neural Networks (CNNs). The goal is to leverage long-range correlations alongside local spatial information to improve the accuracy of depth estimation from monocular images.

Methodology

The authors identify a fundamental limitation in existing CNN-based methods: the restricted receptive field which hampers performance, particularly for distant objects. Unlike CNNs, Vision Transformers (ViTs) can adeptly capture long-range dependencies due to their global receptive field. However, Transformers often lack the ability to model local spatial details, which are crucial for depth estimation tasks.

To address these issues, the proposed architecture integrates:

1. Transformer Branch: Utilizes a Swin Transformer to model long-range dependencies. The Swin Transformer offers hierarchical feature extraction and reduced computational complexity compared to ViTs.
2. Convolution Branch: Incorporates a light-weight ResNet-based encoder to preserve local spatial information.

Hierarchical Aggregation and Heterogeneous Interaction Module (HAHI)

A key innovation in this work is the HAHI module, designed to enhance feature representation and facilitate interaction between heterogeneous feature types from the Transformer and CNN branches. The module accomplishes:

• Hierarchical Aggregation: Improves multi-level feature aggregation using a deformable self-attention mechanism.
• Heterogeneous Interaction: Models the affinity between Transformer and CNN features, enhancing the decoder’s ability to fuse different information types effectively.

Empirical Results

The effectiveness of DepthFormer is demonstrated through extensive experiments on KITTI, NYU-Depth-v2, and SUN RGB-D datasets. Significant performance improvements over state-of-the-art methods were observed, attributed to the novel combination of global context modelling and local detail preservation. On the KITTI depth benchmark, DepthFormer achieved notably competitive results, validating its performance superiority.

Implications and Future Directions

DepthFormer represents a substantial advancement in the methodology for monocular depth estimation. By combining the strengths of both Transformers and CNNs, the approach sets a promising precedent for other computer vision tasks that require both global and local context understanding.

• Theoretical Advancements: Future work may examine the theoretical underpinnings of Transformer-CNN hybrid models, potentially extending this architecture to other domains.
• Scalability and Efficiency: Research could explore optimized attention mechanisms to reduce computational overhead, broadening the applicability of such models in real-time systems.
• Multimodal Learning: Given HAHI’s input-agnostic nature, extending the framework to include multimodal inputs like LiDAR could further enhance robustness and generalization.

In summary, DepthFormer provides not only an empirical improvement in depth estimation but also a flexible and robust framework that could inspire advancements across various domains within AI and computer vision.