3D Scene Geometry Estimation from 360$^\circ$ Imagery: A Survey (2401.09252v1)

Abstract: This paper provides a comprehensive survey on pioneer and state-of-the-art 3D scene geometry estimation methodologies based on single, two, or multiple images captured under the omnidirectional optics. We first revisit the basic concepts of the spherical camera model, and review the most common acquisition technologies and representation formats suitable for omnidirectional (also called 360$^\circ$, spherical or panoramic) images and videos. We then survey monocular layout and depth inference approaches, highlighting the recent advances in learning-based solutions suited for spherical data. The classical stereo matching is then revised on the spherical domain, where methodologies for detecting and describing sparse and dense features become crucial. The stereo matching concepts are then extrapolated for multiple view camera setups, categorizing them among light fields, multi-view stereo, and structure from motion (or visual simultaneous localization and mapping). We also compile and discuss commonly adopted datasets and figures of merit indicated for each purpose and list recent results for completeness. We conclude this paper by pointing out current and future trends.

• The paper presents methodologies for reconstructing 3D scenes from spherical imagery using monocular, stereo, and multi-view approaches integrated with deep learning.
• It identifies challenges like distortion in equirectangular projections and high computational demands in multi-view systems.
• The survey highlights a shift towards learning-based methods that enhance scene reconstruction for immersive VR/AR experiences and reliable autonomous navigation.

Overview of 3D Scene Geometry Estimation from 360° Imagery

360° cameras have revolutionized the way we capture and experience the world. They allow for immersive experiences in virtual reality (VR), augment reality (AR) applications, and have become valuable tools in various industries from real estate to autonomous driving. Understanding the precise 3D structure of captured scenes is crucial for these technologies to advance. This overview sheds light on methodologies for estimating the 3D geometry of scenes from spherical, or 360°, imagery.

Fundamental Concepts and Challenges

Before diving into methods for 3D geometry estimation, it's essential to understand the basics of spherical imaging. Spherical cameras capture light from all directions, encoding it into a 2D format such as the equirectangular projection. However, converting a spherical scene onto a flat image introduces distortions, especially near image poles. These distortions pose challenges for applying traditional image processing and computer vision algorithms, which are generally designed for planar (perspective) images.

Techniques for 3D Scene Reconstruction

Methods to reconstruct 3D scenes from spherical images fall into monocular (single image), stereoscopic (image pairs), and multi-view (multiple images) approaches. For monocular scenes, modern techniques rely heavily on deep learning, which has shown substantial progress despite the challenge of training models with limited data. When two images are available, the disparity between views can be assessed to deduce depth information, addressing occlusions more effectively. Multi-view setups are even more robust, combining the benefits of monocular and stereo methods, promising accurate reconstruction of entire scenes. They are, however, more computationally demanding and may require complex setups like array cameras or sequential captures from a moving camera.

Current Trends and Benchmarks

The field is witnessing a trend towards learning-based methodologies, fueled by the need to extract depth from complex and occlusion-heavy scenes. These methods must be trained on large, varied datasets that contain annotated depth information. There's also a push for standardizing evaluation metrics to fairly compare different algorithms and gauge their performance reliably.

State-of-the-Art Performance

Evaluating the latest techniques reveals that state-of-the-art algorithms are becoming adept at modeling room layouts from single images, though complex scenes with varied depth ranges remain challenging. Current methods excel in predefined environments but require further refinement to handle outdoor scenes and large-scale, diverse datasets.

Concluding Thoughts

360° image-based 3D scene reconstruction is a dynamically growing field with a strong shift towards deep learning solutions. While current methods show impressive potential, challenges like dealing with distortions, demanding computational requirements, and the necessity for extensive data remain. As VR and AR technologies continue to evolve, so will the algorithms that underpin our understanding of the captured scenes, driving innovations across multiple domains.