FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training (2411.02229v2)

Abstract: The field of novel view synthesis from images has seen rapid advancements with the introduction of Neural Radiance Fields (NeRF) and more recently with 3D Gaussian Splatting. Gaussian Splatting became widely adopted due to its efficiency and ability to render novel views accurately. While Gaussian Splatting performs well when a sufficient amount of training images are available, its unstructured explicit representation tends to overfit in scenarios with sparse input images, resulting in poor rendering performance. To address this, we present a 3D Gaussian-based novel view synthesis method using sparse input images that can accurately render the scene from the viewpoints not covered by the training images. We propose a multi-stage training scheme with matching-based consistency constraints imposed on the novel views without relying on pre-trained depth estimation or diffusion models. This is achieved by using the matches of the available training images to supervise the generation of the novel views sampled between the training frames with color, geometry, and semantic losses. In addition, we introduce a locality preserving regularization for 3D Gaussians which removes rendering artifacts by preserving the local color structure of the scene. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance of our method in few-shot novel view synthesis compared to existing state-of-the-art methods.

• The paper presents FewViewGS, a novel method that uses a multi-stage training pipeline to enable few-shot novel view synthesis with optimized 3D Gaussian Splatting.
• It achieves high rendering quality and consistency by incorporating matching-based constraints and locality-preserving regularization, outperforming traditional NeRF variants on PSNR and SSIM.
• The approach reduces reliance on numerous training images and complex pre-trained models, offering practical benefits for real-time virtual and augmented reality applications.

FewViewGS: Novel View Synthesis Through Optimized Gaussian Splatting

The paper under examination presents a novel method for few-shot novel view synthesis (NVS) called FewViewGS, which is grounded in the utilization of 3D Gaussian Splatting without the reliance on pre-trained depth estimation or diffusion models. This research addresses a significant challenge in novel view synthesis methodologies: the requirement for a substantial number of training images to achieve high-quality rendering. Traditional techniques such as Neural Radiance Fields (NeRF) have shown competence in NVS but fall short in real-time applications and when dealing with sparse image inputs. FewViewGS proposes to advance 3D Gaussian Splatting (3DGS) specifically in few-shot scenarios, where traditional methods face significant limitations due to their dependency on multiple views and intensive computational resources.

Overview of FewViewGS Model

FewViewGS bypasses the typical constraints faced by existing methodologies through the introduction of a multi-stage training scheme composed of pre-training, intermediate, and tuning stages. This approach ensures seamless information transfer from known to novel views and minimizes the risk of overfitting to limited input views. A key innovation in this method is the incorporation of matching-based consistency constraints, which enforce coherence of synthesized unseen images without necessitating complex pre-trained models.

The framework harnesses the theoretical underpinnings of Gaussian splatting, wherein each Gaussian entity is explicitly defined and optimized within a radiance field setting. This allows for high-speed rendering and optimizes photometric loss dynamically. FewViewGS further refines Gaussian Splatting with a locality-preserving regularization technique to address typical rendering artifacts and maintain the spatial color consistency of the scene.

Methodological Advantages

FewViewGS differentiates itself from competing models through several technical facets:

1. Robust Multi-Stage Training: By compartmentalizing the learning process into distinct phases, FewViewGS effectively captures the geometry and color details needed for accurate scene rendering from sparse views.
2. Novel View Consistency: This is achieved by deploying a warp-based approach that utilizes matched correspondences between input images to preserve color and geometric integrity. The technique cleverly circumvents common pitfalls in depth prediction and disparity across domains.
3. Locality-Preserving Regularization: This constraint diminishes visual artifacts significantly by maintaining a consistent appearance across locally defined regions within the reconstructed scene.

Numerical Results and Comparisons

FewViewGS demonstrates superiority in rendering quality across several datasets, including DTU, LLFF, and the synthetic Blender dataset, compared to existing techniques such as 3DGS, RegNeRF, and FreeNeRF. Metrics such as PSNR, SSIM, and LPIPS indicate that FewViewGS achieves higher fidelity and perceptual accuracy. Quantitative evaluations reveal that FewViewGS, even with random Gaussian initialization, provides notable increases in rendering precision. The accurate depth estimation further validates its robustness over traditional NeRF variants and other Gaussian Splatting adaptations.

Implications and Future Prospects

The implications of FewViewGS extend to various applications such as virtual and augmented reality, where efficient and high-quality rendering from limited observations is critical. Furthermore, it proposes a pathway toward more efficient 3D scene synthesis that could mitigate the computational overhead associated with traditional methods. Moving forward, there remains a research avenue to specialize FewViewGS for scenes with high texture variability and leverage advanced feature-matching techniques to amplify the model's robustness across diverse datasets.

In conclusion, the FewViewGS paper extends the capabilities of few-shot NVS by integrating Gaussian Splatting within a structured, multi-phase learning strategy. This innovation shows promise not only in computational efficiency but also in achieving state-of-the-art results without the burdens of additional model complexities such as pre-trained networks for depth estimation.