CoR-GS: Sparse-View 3D Gaussian Splatting via Co-Regularization (2405.12110v2)

Abstract: 3D Gaussian Splatting (3DGS) creates a radiance field consisting of 3D Gaussians to represent a scene. With sparse training views, 3DGS easily suffers from overfitting, negatively impacting rendering. This paper introduces a new co-regularization perspective for improving sparse-view 3DGS. When training two 3D Gaussian radiance fields, we observe that the two radiance fields exhibit point disagreement and rendering disagreement that can unsupervisedly predict reconstruction quality, stemming from the randomness of densification implementation. We further quantify the two disagreements and demonstrate the negative correlation between them and accurate reconstruction, which allows us to identify inaccurate reconstruction without accessing ground-truth information. Based on the study, we propose CoR-GS, which identifies and suppresses inaccurate reconstruction based on the two disagreements: (1) Co-pruning considers Gaussians that exhibit high point disagreement in inaccurate positions and prunes them. (2) Pseudo-view co-regularization considers pixels that exhibit high rendering disagreement are inaccurate and suppress the disagreement. Results on LLFF, Mip-NeRF360, DTU, and Blender demonstrate that CoR-GS effectively regularizes the scene geometry, reconstructs the compact representations, and achieves state-of-the-art novel view synthesis quality under sparse training views.

• The paper introduces co-regularization by aligning dual radiance fields to suppress reconstruction errors in sparse view scenarios.
• It employs co-pruning to remove inaccurately positioned Gaussians and pseudo-view co-regularization to enforce consistent rendering.
• Experimental results on benchmarks like LLFF and DTU demonstrate improved PSNR, SSIM, and reduced LPIPS, confirming enhanced reconstruction quality.

Exploring CoR-GS: Enhancing Sparse-View 3D Gaussian Splatting via Co-Regularization

Overview

3D Gaussian Splatting (3DGS) has gained prominence as a technique for generating a radiance field composed of 3D Gaussians that represent a scene. However, 3DGS often suffers from overfitting when trained with a limited number of views, leading to suboptimal reconstruction quality. A paper proposes a new method called CoR-GS, which introduces co-regularization to address this issue.

Key Concepts

Before diving into the paper, let's break down some key concepts:

• 3D Gaussian Splatting (3DGS): This technique uses 3D Gaussians to represent volumetric data and perform rendering.
• Radiance Field: A representation of how light interacts with a 3D scene, used to generate new views.
• Sparse Views: A scenario where only a limited number of training images are available to reconstruct the scene.
• Co-Regularization: A method to ensure consistent behavior between two simultaneously trained models to improve accuracy.

Point and Rendering Disagreement

The central idea of CoR-GS revolves around the concepts of point disagreement and rendering disagreement between two 3D Gaussian radiance fields trained on the same sparse views:

• Point Disagreement: Measures the positional differences of corresponding Gaussians in the two sets, which can be quantified using metrics like Fitness and RMSE.
• Rendering Disagreement: Evaluates rendering inconsistencies, using metrics such as PSNR and SSIM to compare rendered images and depth maps between the two radiance fields.

The paper finds that larger disagreements often correspond to areas of inaccurate reconstruction. By identifying these discrepancies, the method aims to suppress inaccurate parts of the reconstruction.

Co-Pruning and Pseudo-View Co-Regularization

CoR-GS employs two main techniques to achieve its goals:

1. Co-Pruning:
   • This process identifies and removes Gaussians in positions with high point disagreement, which typically indicate inaccuracy.
   • It involves matching Gaussians from each radiance field and pruning those without close matches in the opposite field.
2. Pseudo-View Co-Regularization:
   • This technique focuses on rendering disagreement by sampling pseudo views (interpolated from existing views) to enforce consistency.
   • It blends rendered images from the pseudo views with a combination of L1 losses and Structural Similarity (SSIM) metrics to regularize the learning process.

Practical Implications and Results

The CoR-GS approach has been tested on several benchmarks, including LLFF, Mip-NeRF360, DTU, and Blender datasets, demonstrating its effectiveness:

• Quantitative Performance: CoR-GS consistently achieves higher PSNR, SSIM, and lower LPIPS scores across various sparsely-viewed scenes.
• Efficiency: Despite adding some training overhead, CoR-GS maintains efficiency at inference time, thanks to its compact representations.

For intermediate data scientists, this indicates that CoR-GS could become a critical tool in scenarios where data is limited, enabling better reconstructions with fewer images.

Future Directions

While CoR-GS shows promise, the paper opens up avenues for further research:

• Integration with Other Techniques: Combining co-regularization with other forms of supervision, like depth maps, could further enhance performance.
• Application in Dynamic Scenes: Investigating how CoR-GS performs on dynamic or temporal data could yield interesting insights.

Overall, CoR-GS offers a robust method for improving 3DGS in sparse-view conditions, and its principles could be extended to various AI applications in 3D reconstruction and beyond.