InstantSplat: Sparse-view SfM-free Gaussian Splatting in Seconds (2403.20309v4)

Abstract: While neural 3D reconstruction has advanced substantially, it typically requires densely captured multi-view data with carefully initialized poses (e.g., using COLMAP). However, this requirement limits its broader applicability, as Structure-from-Motion (SfM) is often unreliable in sparse-view scenarios where feature matches are limited, resulting in cumulative errors. In this paper, we introduce InstantSplat, a novel and lightning-fast neural reconstruction system that builds accurate 3D representations from as few as 2-3 images. InstantSplat adopts a self-supervised framework that bridges the gap between 2D images and 3D representations using Gaussian Bundle Adjustment (GauBA) and can be optimized in an end-to-end manner. InstantSplat integrates dense stereo priors and co-visibility relationships between frames to initialize pixel-aligned geometry by progressively expanding the scene avoiding redundancy. Gaussian Bundle Adjustment is used to adapt both the scene representation and camera parameters quickly by minimizing gradient-based photometric error. Overall, InstantSplat achieves large-scale 3D reconstruction in mere seconds by reducing the required number of input views. It achieves an acceleration of over 20 times in reconstruction, improves visual quality (SSIM) from 0.3755 to 0.7624 than COLMAP with 3D-GS, and is compatible with multiple 3D representations (3D-GS, 2D-GS, and Mip-Splatting).

• The paper introduces InstantSplat, a unified framework that integrates 3D Gaussian splatting with dense stereo priors to address sparse-view and pose-free novel view synthesis challenges.
• It employs a two-stage process with Coarse Geometric Initialization for fast scene and camera estimation followed by Fast 3D-Gaussian Optimization to refine parameters efficiently.
• Experimental results on Tanks & Temples demonstrate a 32% increase in SSIM and an 80% reduction in ATE, highlighting significant improvements in rendering quality and pose accuracy.

InstantSplat: Efficient and Unified Framework for Sparse-View 3D Reconstruction and Novel View Synthesis

Introduction

The newfound framework, named InstantSplat, introduces a novel methodology for addressing the challenges in novel view synthesis (NVS) under sparse-view and pose-free conditions. Through the integration of 3D Gaussian Splatting and Dense Stereo Priors, InstantSplat establishes itself as a potent solution for reconstructing 3D scenes and synthesizing novel views with high fidelity. The framework distinguishes itself by significantly improving both pose estimation accuracy and rendering quality, backed by strong numerical results on the Tanks and Temples datasets. The processes culminate in a robust system that can operate within one minute for large-scale scenes, marking a notable advancement in the field of 3D computer vision.

Key Contributions

• InstantSplat innovates by leveraging 3D Gaussian Splatting with dense stereo priors derived from an end-to-end dense stereo model (DUSt3R), effectively tackling sparse-view and pose-free challenges in NVS.
• The framework encompasses two main components: a Coarse Geometric Initialization (CGI) module for rapid preliminary scene structure and camera parameter estimation, and a Fast 3D-Gaussian Optimization (F-3DGO) module for joint optimization of 3D Gaussian attributes and initialized poses.
• Demonstrated improvements include a 32% increase in SSIM and an 80% reduction in Absolute Trajectory Error (ATE) on the Tanks & Temples datasets compared to existing methods, evidencing its capability to maintain high rendering quality and accurate pose estimation in sparse and unconditioned scenarios.

Methodology Overview

Coarse Geometric Initialization (CGI)

The CGI module harnesses the dense stereo model, DUSt3R, to predict globally aligned 3D point maps from sparse-view images. This alignment furnishes an initial geometric and photographic context that facilitates the rapid estimation of preliminary scene structures and camera parameters.

Fast 3D-Gaussian Optimization (F-3DGO)

Following CGI, the F-3DGO module employs these initial estimates to refine the 3D Gaussian attributes and camera poses further. It implements pose regularization, substantially enhancing the final pose accuracy and rendering quality through an efficient optimization process.

Experimental Insights

Extensive evaluations on the outdoor Tanks & Temples datasets underscore InstantSplat's superiority in sparse-view and pose-free scenarios. The method not only significantly outperforms existing pose-free methods in rendering quality but also showcases remarkable improvements in pose estimation accuracy.

Theoretical and Practical Implications

On a theoretical level, InstantSplat presents a novel approach to NVS tasks by combining explicit 3D representation with pose priors, diverging from the dependence on dense data coverage or prior knowledge of camera parameters. Practically, the method's efficiency and effectiveness in handling real-world scenarios indicate its potential applicability in areas such as digital twin construction, augmented reality, and beyond.

Future Directions

The current landscape of NVS under sparse-view conditions suggests a promising direction for future research to explore the integration of machine learning techniques with explicit 3D representations further. Developments in end-to-end systems capable of reconstructing and rendering scenes from extremely sparse and unconditioned inputs could revolutionize 3D content creation and visualization technologies.

Summary

InstantSplat represents a significant leap towards solving the long-standing challenges in novel view synthesis, specifically in sparse-view and pose-free settings. By proficiently merging the capabilities of dense stereo models with 3D Gaussian Splatting, it offers a fast, accurate, and practicable solution for 3D scene reconstruction and rendering, paving the way for next-generation 3D vision applications.