UniFuse: Unidirectional Fusion for 360$^{\circ}$ Panorama Depth Estimation (2102.03550v2)

Abstract: Learning depth from spherical panoramas is becoming a popular research topic because a panorama has a full field-of-view of the environment and provides a relatively complete description of a scene. However, applying well-studied CNNs for perspective images to the standard representation of spherical panoramas, i.e., the equirectangular projection, is suboptimal, as it becomes distorted towards the poles. Another representation is the cubemap projection, which is distortion-free but discontinued on edges and limited in the field-of-view. This paper introduces a new framework to fuse features from the two projections, unidirectionally feeding the cubemap features to the equirectangular features only at the decoding stage. Unlike the recent bidirectional fusion approach operating at both the encoding and decoding stages, our fusion scheme is much more efficient. Besides, we also designed a more effective fusion module for our fusion scheme. Experiments verify the effectiveness of our proposed fusion strategy and module, and our model achieves state-of-the-art performance on four popular datasets. Additional experiments show that our model also has the advantages of model complexity and generalization capability.The code is available at https://github.com/alibaba/UniFuse-Unidirectional-Fusion.

• The paper’s main contribution is a unidirectional fusion strategy that integrates cubemap features into equirectangular representations to enhance depth prediction.
• It introduces the CEE fusion module, employing residual modulation and Squeeze-and-Excitation blocks to effectively address distortions and discontinuities.
• Empirical results across four datasets show UniFuse significantly reduces Abs Rel error and improves accuracy without increasing model complexity.

Overview of UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation

The paper under discussion introduces a novel approach named "UniFuse" aimed at improving depth estimation from 360° panoramic images. This research targets a prominent challenge in the field of 3D reconstruction—accurate depth estimation from spherical images with full field-of-view. Traditional methods using equirectangular projection (ERP) or cubemap projection (CMP) entail inherent limitations such as distortion towards poles and discontinuities at face edges. UniFuse proposes a fusion-based framework that unidirectionally integrates features from both ERP and CMP to mitigate these issues while optimizing computational efficiency.

Key Methodological Insights

1. Panoramic Representations: UniFuse makes use of both ERP and CMP representations to leverage their unique advantages—ERP’s complete scene coverage and CMP’s distortion-free characteristics. It provides a novel fusion framework that unidirectionally feeds cubemap features into equirectangular features only during the decoding stage.
2. Unidirectional Fusion Approach: Unlike the bidirectional approach seen in methods like BiFuse, the unidirectional strategy in UniFuse enhances computational speed and avoids unnecessary complexity. By focusing on the decoding phase, it better supports the enhancement of equirectangular depth prediction.
3. CEE Fusion Module: At the core of UniFuse is the CEE (Cubemap to Enhance Equirectangular) module, which applies residual modulation to cubemap features to address discontinuities. A residual block inspired by ResNet architecture serves to fill inconsistencies between cubemap face boundaries. Subsequently, a Squeeze-and-Excitation block recalibrates feature responses channel-wise, further fine-tuning the fusion of features.

Numerical Results and Claims

This paper presents empirical results on four widespread datasets: Matterport3D, Stanford2D3D, 3D60, and PanoSUNCG, demonstrating that UniFuse consistently achieves state-of-the-art performance. Notably, on the Matterport3D dataset, UniFuse surpassed BiFuse by substantially lowering the Abs Rel error from 0.2048 to 0.1063 and enhancing the $\delta < 1.25$ accuracy metric by over 5%. Without inflating model complexity, it establishes robust baseline performance with an exceptional fit across various testing scenarios, heralding substantial improvements over prior benchmarks.

Implications and Future Directions

The implications of UniFuse extend into critical applications such as robotic navigation and AR/VR systems where accurate depth perception is crucial. It offers a robust framework capable of generating more reliable and comprehensive 3D reconstructions without the burdens of excessive computational overhead. Moving forward, optimizing this approach for mobile deployment seems promising due to its markedly low parameter count and inference latency, as evidenced by further enhancements when integrated with lightweight architectures like MobileNetV2.

Additionally, exploring the adaptability of UniFuse to integrate additional visual cues or to consider temporal consistencies in dynamic environments could further broaden its application scope. Real-world deployment would benefit from further empirical validations in uncontrolled settings to ascertain its robustness against varying lighting and motion conditions, critical for mobile platforms and augmented reality applications.

In summary, UniFuse represents a significant step toward efficient and precise depth estimation in panoramic imaging. By critically addressing methodological challenges inherent to ERP and CMP, and advancing the efficacy of feature fusion strategies, it demonstrates a promising pathway for the future of 3D reconstruction technologies.