Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction (2502.09563v2)

Abstract: In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion and 3D Gaussian representations, enabling accurate and efficient scene reconstruction. In particular, our technique enables high-quality scene reconstruction from Large field-of-view (FOV) imagery taken with wide-angle lenses, allowing the scene to be modeled from a smaller number of images. Our approach introduces a novel method for modeling complex lens distortions using a hybrid network that combines invertible residual networks with explicit grids. This design effectively regularizes the optimization process, achieving greater accuracy than conventional camera models. Additionally, we propose a cubemap-based resampling strategy to support large FOV images without sacrificing resolution or introducing distortion artifacts. Our method is compatible with the fast rasterization of Gaussian Splatting, adaptable to a wide variety of camera lens distortion, and demonstrates state-of-the-art performance on both synthetic and real-world datasets.

• The paper introduces a self-calibrating Gaussian splatting method that achieves high-quality 3D scene reconstruction directly from uncalibrated, highly distorted wide-angle images.
• The method utilizes a hybrid neural network to accurately model complex lens distortions and employs a cubemap-based resampling strategy for effective handling of large fields of view.
• Experimental results demonstrate superior performance in reconstruction quality on diverse datasets, enabling practical applications like faster data acquisition for robotics and virtual reality.

Overview of "Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction"

The paper "Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction" presents a novel methodology to enable high-quality 3D scene reconstruction using wide-angle imagery. This work specifically addresses challenges associated with reconstructing scenes from highly distorted wide-angle or fisheye lens captures without pre-calibration. The authors introduce a method that not only self-calibrates camera parameters during the reconstruction process but also effectively models lens distortions that are often nonlinear and complex.

Methodological Contributions

The authors propose a multi-faceted approach that integrates several novel techniques to solve the problem of scene reconstruction from distorted wide-angle images:

1. Hybrid Neural Lens Distortion Modeling: The paper introduces a sophisticated lens distortion modeling approach using a hybrid neural network structure. This combines invertible residual networks with explicit grid representations to exploit the strengths of both parametric and non-parametric models. Such a design allows accurate modeling of complex lens distortions, improving rendering quality over traditional distortion models that cannot fully capture the nuances of fisheye optics.
2. Cubemap-Based Resampling: To overcome the issue of resolution sacrifice typical of single planar projections, the authors propose using a cubemap-based resampling strategy. This allows for the even distribution of image data, maintaining consistency in pixel density and reducing distortion artifacts across the large field of view images.
3. Differentiable Gaussian Splatting Pipeline: The proposed method is built on an efficient differentiable rendering framework using Gaussian splatting. This approach is compatible with a wide range of camera models and has shown state-of-the-art performance on both synthetic and real datasets.

Experimental Results

The paper provides an extensive evaluation of the proposed method. It includes comparisons with existing techniques such as standard Gaussian splatting and other self-calibrating radiance field methods. The results demonstrate significant improvements in metrics like PSNR and SSIM across various datasets, including complex real-world scenes captured with uncalibrated wide-angle cameras.

The authors conduct experiments on synthetic datasets from the Mitsuba renderer and real-world datasets to evaluate the efficacy of their approach. The proposed method consistently outperforms traditional methods, especially under extreme lens distortion conditions. This validates the robustness of the hybrid distortion field in handling diverse distortion types without pre-calibration.

Implications and Future Directions

The advancements presented in this work have far-reaching implications for applications in robotics, virtual reality, and other areas that require efficient and accurate 3D scene reconstruction from less conventional imaging setups. The ability to utilize wide-angle images effectively enables faster data acquisition and reconstruction with fewer images, thus providing significant practical benefits.

Future research could further improve the modeling of various optical effects such as vignetting and chromatic aberration not explicitly addressed in this work. Extending the pipeline to handle entrance pupil shift in fisheye lenses represents another potential avenue for enhancing the robustness of scene reconstructions in all-encompassing field of view scenarios.

The flexibility and adaptability of the proposed method suggest promising integration with other advanced 3D reconstruction and machine learning frameworks, potentially driving further innovations in the field. As computational capabilities and neural modeling techniques continue to evolve, the approach detailed in the paper could serve as a foundational element for the development of even more refined and capable 3D scene reconstruction processes.