Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set (2410.14189v1)

Abstract: It is vital to infer a signed distance function (SDF) in multi-view based surface reconstruction. 3D Gaussian splatting (3DGS) provides a novel perspective for volume rendering, and shows advantages in rendering efficiency and quality. Although 3DGS provides a promising neural rendering option, it is still hard to infer SDFs for surface reconstruction with 3DGS due to the discreteness, the sparseness, and the off-surface drift of 3D Gaussians. To resolve these issues, we propose a method that seamlessly merge 3DGS with the learning of neural SDFs. Our key idea is to more effectively constrain the SDF inference with the multi-view consistency. To this end, we dynamically align 3D Gaussians on the zero-level set of the neural SDF using neural pulling, and then render the aligned 3D Gaussians through the differentiable rasterization. Meanwhile, we update the neural SDF by pulling neighboring space to the pulled 3D Gaussians, which progressively refine the signed distance field near the surface. With both differentiable pulling and splatting, we jointly optimize 3D Gaussians and the neural SDF with both RGB and geometry constraints, which recovers more accurate, smooth, and complete surfaces with more geometry details. Our numerical and visual comparisons show our superiority over the state-of-the-art results on the widely used benchmarks.

• The paper introduces neural pulling to dynamically align 3D Gaussians with the zero-level set, significantly improving SDF inference.
• It integrates RGB and geometric constraints to jointly optimize 3D Gaussians and neural SDFs for more accurate surface reconstructions.
• Empirical results demonstrate superior performance and smoother, more complete 3D reconstructions compared to state-of-the-art methods.

Neural Signed Distance Function Inference through Splatting 3D Gaussians Pulled on Zero-Level Set

This paper presents a novel approach to inferring neural signed distance functions (SDFs) for multi-view 3D surface reconstruction, leveraging 3D Gaussian splatting (3DGS). The key challenge addressed by the authors is the discreteness, sparseness, and off-surface drift inherent in 3D Gaussians, which complicate their effective use in surface reconstruction. The proposed method seeks to merge the strengths of 3DGS with neural SDF learning, particularly focusing on enhancing multi-view consistency constraints.

Methodology

The authors introduce a technique that aligns 3D Gaussians dynamically with the zero-level set of a neural SDF using a process termed "neural pulling." This process involves rendering the aligned 3D Gaussians using differentiable rasterization, while concurrently updating the neural SDF by pulling neighboring space to the newly aligned 3D Gaussians. This segmentation refines the signed distance field around the surface in a progressive manner.

The methodology benefits from both RGB and geometric constraints, optimizing 3D Gaussians and the neural SDF collectively. The differentiable pulling operation, which employs predicted signed distances and gradients from the neural SDF, is pivotal in imposing both RGB and geometry constraints on 3D Gaussians.

Results

Empirical evaluations demonstrate the method's superiority over existing state-of-the-art techniques on widely recognized benchmarks. The paper presents numerical and visual comparisons that substantiate the method's effectiveness in achieving more accurate, smooth, and complete surface reconstructions, with enhanced geometric detail.

Implications and Future Directions

The integration of neural SDFs with 3DGS provides an efficient alternative to traditional neural radiance fields (NeRFs) by circumventing the computational burden of NeRFs' stochastic sampling along rays. This presents a potential pathway for improving both the quality and speed of neural rendering processes.

Future work could explore extending this methodology to handle scenes with more complex surface geometries or incorporating more advanced neural architectures to improve SDF inference capabilities. Additionally, the technique could be adapted for dynamic scene reconstruction, potentially broadening its applicability in various computer vision applications.

Conclusion

This paper offers a significant contribution to the field of neural rendering and 3D reconstruction by effectively addressing the challenges posed by 3D Gaussian discreteness and sparseness. The novel integration of neural SDF inference through 3D Gaussian splatting introduces new perspectives on optimizing multi-view consistency, thereby enhancing the practical utility of neural-based surface reconstruction technologies.