360-GS: Layout-guided Panoramic Gaussian Splatting For Indoor Roaming (2402.00763v1)

Abstract: 3D Gaussian Splatting (3D-GS) has recently attracted great attention with real-time and photo-realistic renderings. This technique typically takes perspective images as input and optimizes a set of 3D elliptical Gaussians by splatting them onto the image planes, resulting in 2D Gaussians. However, applying 3D-GS to panoramic inputs presents challenges in effectively modeling the projection onto the spherical surface of ${360^\circ}$ images using 2D Gaussians. In practical applications, input panoramas are often sparse, leading to unreliable initialization of 3D Gaussians and subsequent degradation of 3D-GS quality. In addition, due to the under-constrained geometry of texture-less planes (e.g., walls and floors), 3D-GS struggles to model these flat regions with elliptical Gaussians, resulting in significant floaters in novel views. To address these issues, we propose 360-GS, a novel $360^{\circ}$ Gaussian splatting for a limited set of panoramic inputs. Instead of splatting 3D Gaussians directly onto the spherical surface, 360-GS projects them onto the tangent plane of the unit sphere and then maps them to the spherical projections. This adaptation enables the representation of the projection using Gaussians. We guide the optimization of 360-GS by exploiting layout priors within panoramas, which are simple to obtain and contain strong structural information about the indoor scene. Our experimental results demonstrate that 360-GS allows panoramic rendering and outperforms state-of-the-art methods with fewer artifacts in novel view synthesis, thus providing immersive roaming in indoor scenarios.

• The paper introduces a 360° Gaussian Splatting algorithm that projects 3D Gaussians onto spherical surfaces for effective panoramic modeling.
• It leverages layout-guided initialization and regularization, using room layout cues to enhance 3D structure recovery from sparse panoramic inputs.
• Empirical evaluations on Matterport3D demonstrate significant improvements in PSNR, SSIM, and LPIPS metrics over conventional NeRF and 3D-GS methods.

Overview of 360-GS: Layout-guided Panoramic Gaussian Splatting for Indoor Roaming

The paper presents 360-GS, a layout-guided panoramic Gaussian splatting approach that enhances 3D Gaussian Splatting (3D-GS) for novel view synthesis using panoramas. While 3D-GS has shown potential in real-time, photo-realistic renderings with perspective images, it faces challenges when applied to panoramic inputs. The innovative 360-GS method addresses these challenges, providing effective modeling of projections onto spherical surfaces inherent in 360-degree imagery. This paper introduces key advancements such as the $360^{\circ}$ Gaussian splatting technique and the utilization of layout priors within panoramas to deliver high-quality novel view synthesis with fewer artifacts.

Methodological Contributions

There are several critical components in the proposed 360-GS approach:

1. $360^{\circ}$ Gaussian Splatting Algorithm: Traditional 3D Gaussian splatting techniques rely on perspective projections that do not adapt well to panoramic images due to spatial distortions. The paper proposes a two-step projection: first, mapping 3D Gaussians onto the tangent plane of a unit sphere; second, transitioning these to spherical projections. This strategy allows effective panorama modeling while maintaining the computational efficiency necessary for real-time applications.
2. Layout-guided Initialization and Regularization: The integration of layout priors considerably enhances the initialization of 3D Gaussians. This involves using structural cues like room layouts to provide stronger starting geometry, reducing the ambiguity in learning 3D structures from sparse 2D inputs. Additionally, layout-guided regularization retains the structural coherence by minimizing the deviation of optimized 3D Gaussians from expected positions derived from panoramic layout information.

Experimental Evaluation

The authors demonstrate the efficacy of 360-GS through empirical evaluations on real-world datasets from the Matterport3D repository, using both sparse (4-view) and dense (32-view) panoramic inputs. The proposed method was benchmarked against state-of-the-art methods such as NeRF variants and 3D-GS. In scenarios with sparse panoramic inputs, 360-GS showed significant performance improvements in terms of PSNR, SSIM, and LPIPS metrics, indicating superior visual fidelity and structural consistency. The method also maintained competitive results with increased input views, corroborating its scalability and robustness across various viewing conditions.

Implications and Future Directions

The integration of structural layout priors into 3D Gaussian representations signifies a crucial advancement in the domain of panoramic view synthesis. By addressing the limitations in projection modeling on spherical surfaces and leveraging readily available panoramic layout information, 360-GS sets a new standard for high-quality, real-time panorama rendering.

Future research could extend 360-GS to encompass broader scene representations, incorporate dynamic scene elements, or integrate with machine learning models for further optimization. Furthermore, exploring the applications of this approach in virtual reality (VR) and augmented reality (AR) could unfold new opportunities in immersive media. The paper's advancements are likely to spark continued interest in adaptive rendering solutions for panoramic imagery, particularly in environments where reducing artifacts and improving real-time efficacy are of paramount importance.