Pixel-GS: Density Control with Pixel-aware Gradient for 3D Gaussian Splatting (2403.15530v1)

Abstract: 3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis results while advancing real-time rendering performance. However, it relies heavily on the quality of the initial point cloud, resulting in blurring and needle-like artifacts in areas with insufficient initializing points. This is mainly attributed to the point cloud growth condition in 3DGS that only considers the average gradient magnitude of points from observable views, thereby failing to grow for large Gaussians that are observable for many viewpoints while many of them are only covered in the boundaries. To this end, we propose a novel method, named Pixel-GS, to take into account the number of pixels covered by the Gaussian in each view during the computation of the growth condition. We regard the covered pixel numbers as the weights to dynamically average the gradients from different views, such that the growth of large Gaussians can be prompted. As a result, points within the areas with insufficient initializing points can be grown more effectively, leading to a more accurate and detailed reconstruction. In addition, we propose a simple yet effective strategy to scale the gradient field according to the distance to the camera, to suppress the growth of floaters near the camera. Extensive experiments both qualitatively and quantitatively demonstrate that our method achieves state-of-the-art rendering quality while maintaining real-time rendering speed, on the challenging Mip-NeRF 360 and Tanks & Temples datasets.

Pixel-GS: Enhancing 3D Gaussian Splatting with Pixel-aware Gradient for Improved Density Control in Novel View Synthesis

Introduction

In the exploration of point-based radiance fields for real-time rendering and novel view synthesis (NVS), the 3D Gaussian Splatting (3DGS) method has marked its significance by offering an explicit point-based representation of 3D scenes. Despite its advancements in rendering quality and speed, the method's performance is heavily tied to the quality of the initially generated point cloud. The inadequacy in the number of initializing points leads to blurring and needle-like artifacts, particularly in areas with sparse initial points. This challenge roots in the growth condition applied during the point cloud optimization phase, which does not facilitate sufficient growth for large Gaussians visible across multiple viewpoints.

Method

The introduced methodology, Pixel-GS, presents a novel solution by considering the pixel coverage of each Gaussian in the growth condition calculation. By leveraging a weighted average of the gradients involving pixel coverage information, the approach significantly prompts the growth of large Gaussians. This strategic optimization addresses the artifacts' issue by enhancing point growth in areas poorly initialized. Additionally, the paper proposes a gradient scaling strategy based on the distance to the camera, aiming to suppress unwanted growth near the camera, effectively mitigating floater artifacts.

Key Contributions:

• Pixel-aware Gradient Averaging: By employing a weighted averaging method that prioritizes pixel coverage by each Gaussian, the mechanism efficiently grows points in areas with insufficient initializing points.
• Gradient Field Scaling: This strategy scales the gradient field responsible for point growth, significantly diminishing the occurrence of floaters, thus improving scene fidelity near the camera viewpoint.
• Robust Performance on Challenging Datasets: The proposed method is rigorously evaluated across challenging datasets, including Mip-NeRF 360 and Tanks & Temples, showcasing superior rendering quality coupled with maintaining real-time rendering capabilities.

Experiments and Results

Extensive experimentation validates the proficiency of Pixel-GS over conventional 3DGS, particularly highlighting its robustness in improving rendering quality across various scenes with complex textural and geometric properties. Quantitative results emphasize a notable enhancement in rendering metrics - PSNR, SSIM, and especially LPIPS, reflecting considerable perceptual improvement in synthesized views.

Observations:

• Effective Addressal of Initial Point Cloud Sparsity: Pixel-GS demonstrated a pronounced ability to generate additional points effectively in areas lacking sufficient initializing points, contributing to ameliorating blurring and artifact issues.
• Balanced Point Growth and Resource Utilization: Despite slightly increased memory usage, the growth of points is strategically focused on areas necessitating densification, ensuring an optimal balance between rendering quality and resource expenditure.

Practical Implications and Future Perspectives

The methodology's ability to enhance point cloud representation directly translates to improved NVS quality, particularly beneficial for applications demanding high-fidelity 3D scene reconstructions, such as virtual reality and augmented reality environments. Looking forward, the principles underlying Pixel-GS could inspire further investigations into adaptive density control mechanisms, possibly extending beyond point-based radiance fields to other scene representations.

Concluding Remarks

Pixel-GS introduces a pivotal advancement in optimizing point cloud density control for 3D Gaussian Splatting, addressing inherent limitations tied to the initial point cloud quality. Through innovative pixel-aware gradient computation and strategic gradient field scaling, the approach commendably enhances the rendering quality and robustness of NVS, setting a promising avenue for future research in real-time 3D scene rendering and synthesis.