MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields (2410.11394v1)

Abstract: Radiance fields represented by 3D Gaussians excel at synthesizing novel views, offering both high training efficiency and fast rendering. However, with sparse input views, the lack of multi-view consistency constraints results in poorly initialized point clouds and unreliable heuristics for optimization and densification, leading to suboptimal performance. Existing methods often incorporate depth priors from dense estimation networks but overlook the inherent multi-view consistency in input images. Additionally, they rely on multi-view stereo (MVS)-based initialization, which limits the efficiency of scene representation. To overcome these challenges, we propose a view synthesis framework based on 3D Gaussian Splatting, named MCGS, enabling photorealistic scene reconstruction from sparse input views. The key innovations of MCGS in enhancing multi-view consistency are as follows: i) We introduce an initialization method by leveraging a sparse matcher combined with a random filling strategy, yielding a compact yet sufficient set of initial points. This approach enhances the initial geometry prior, promoting efficient scene representation. ii) We develop a multi-view consistency-guided progressive pruning strategy to refine the Gaussian field by strengthening consistency and eliminating low-contribution Gaussians. These modular, plug-and-play strategies enhance robustness to sparse input views, accelerate rendering, and reduce memory consumption, making MCGS a practical and efficient framework for 3D Gaussian Splatting.

• The paper introduces a novel MCGS framework that improves multiview consistency using a sparse initializer and feature-driven progressive pruning.
• It employs edge-aware depth regularization to mitigate artifacts and maintain geometric integrity in low-texture regions.
• Experimental results demonstrate higher PSNR, SSIM, and rendering speed, highlighting its potential for real-time AR/VR and robotics applications.

Overview of MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields

The paper "MCGS: Multiview Consistency Enhancement for Sparse-View 3D Gaussian Radiance Fields" introduces a novel methodology to enhance the multiview consistency in 3D Gaussian Radiance Fields (3DGS) when operating under sparse view conditions. The core challenge addressed by this work is the performance degradation that 3DGS experiences due to insufficient multi-view constraints when the input views are sparse. This paper presents MCGS, a framework that employs innovative strategies for initialization and optimization, improving the robustness and efficiency of 3D Gaussian Splatting for sparse novel view synthesis.

Key Innovations

1. Sparse Initializer: The authors introduce a sparse initializer leveraging a pre-trained sparse matching network, LightGlue, to extract initial correspondences from available views. The paired points are averaged to generate initial points. This approach circumvents the density requirement imposed by conventional methods such as Structure-from-Motion (SFM) and Multi-View Stereo (MVS). A random filling strategy is employed to further populate the scene, ensuring that the initialized geometry is comprehensive yet minimally intrusive.
2. Multi-view Consistency Guided Progressive Pruning: The paper proposes a pruning strategy utilizing high-level visual features from DINOv2 to incrementally refine the Gaussian representation. This strategy helps in identifying Gaussians with weak color consistency, thereby focusing optimization on geometrically relevant points, effectively refining the representation by eliminating low-value Gaussians.
3. Edge-Aware Depth Regularization: To address geometric voids that may arise from pruning, the method employs edge-aware depth regularization, enhancing depth map continuity, mitigating artifacts at object boundaries, and maintaining geometric integrity, particularly in low-texture regions.

Experimental Results

The MCGS framework demonstrates considerable improvements across several datasets including LLFF, Blender, and DTU. The method achieves higher PSNR and SSIM values, and reduced LPIPS, demonstrating enhanced visual quality and consistency. Notably, the proposed techniques considerably reduce the number of Gaussian primitives required, thereby boosting rendering speed and decreasing memory consumption. The efficiency assessment highlights that MCGS outperforms previous methods in rendering speed, achieving real-time performance while maintaining visual fidelity.

Implications and Future Directions

The MCGS framework addresses critical limitations in novel view synthesis under sparse input conditions by introducing multiview consistency constraints directly into the initialization and optimization processes. The implications for practical applications are significant, particularly in fields such as AR/VR and robotics where real-time and photorealistic rendering with minimal input is crucial.

The integration of feature-based pruning and depth regularization paves the way for future work to explore further enhancements in efficiency and scene representation accuracy. Potential developments could involve adaptive learning mechanisms that dynamically adjust parameters based on input scene complexity or the combination of MCGS with generative methods for improved geometric priors.

In conclusion, MCGS represents a significant step towards achieving efficient and consistent 3D Gaussian radiance fields from sparse views. The proposed approach not only enhances the performance of existing 3DGS architectures but also sets a foundation for further advancements in efficient scene representation and novel view synthesis.