DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading (2404.09412v2)

Abstract: Reconstructing and editing 3D objects and scenes both play crucial roles in computer graphics and computer vision. Neural radiance fields (NeRFs) can achieve realistic reconstruction and editing results but suffer from inefficiency in rendering. Gaussian splatting significantly accelerates rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting utilizes a single Spherical Harmonic (SH) function to model both texture and lighting, limiting independent editing capabilities of these components. Recently, attempts have been made to decouple texture and lighting with the Gaussian splatting representation but may fail to produce plausible geometry and decomposition results on reflective scenes. Additionally, the forward shading technique they employ introduces noticeable blending artifacts during relighting, as the geometry attributes of Gaussians are optimized under the original illumination and may not be suitable for novel lighting conditions. To address these issues, we introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. To achieve successful decoupling, we model the illumination with a learnable environment map and define additional attributes such as texture parameters and normal direction on Gaussians, where the normal is distilled from a jointly trained signed distance function. More importantly, we apply deferred shading, resulting in more realistic relighting effects compared to previous methods. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view synthesis and editing tasks.

• The paper introduces DeferredGS, a method that decouples geometry, texture, and lighting for independent editing using a normal distillation module.
• It employs deferred shading to overcome blending artifacts, enabling real-time rendering at approximately 30 FPS on an 800×800 resolution with a 3090 GPU.
• Experimental results demonstrate superior novel view synthesis and editing capabilities compared to state-of-the-art methods.

DeferredGS: Enabling Editable Gaussian Splatting for Efficient Scene Reconstruction and Editing

Introduction to the Research

Recent advancements in reconstructing 3D scenes from multi-view images have demonstrated significant potential in computer graphics and computer vision. Among various approaches, Neural Radiance Fields (NeRFs) have gained popularity for their ability to synthesize novel views. However, NeRF-based methods are not without drawbacks, primarily due to their inefficient rendering process, which heavily relies on dense sampling. An alternate method, Gaussian Splatting, offers a solution to this inefficiency by approximating a scene with 3D Gaussians, thus achieving real-time rendering capabilities. Despite its advantages in rendering performance, Gaussian Splatting's lack of support for editing has limited its application. Addressing this gap, the paper introduces DeferredGS, a novel approach that enhances Gaussian Splatting with decoupled geometry, texture, and lighting, enabling efficient editing and rendering.

Key Contributions

• Introduction of DeferredGS: A decoupled Gaussian Splatting representation that facilitates the independent editing of geometry, texture, and lighting. This approach also includes a normal distillation module to improve geometry fidelity.
• Deferred Shading with Gaussian Splatting: The authors' implementation is the first to apply deferred shading to Gaussian Splatting, effectively overcoming blending artifacts encountered in previous methods.
• Promising Experimental Results: DeferredGS not only demonstrates improved decomposition and editing outcomes but also maintains real-time rendering performance at approximately 30 frames per second for an $800 \times 800$ resolution on a single 3090 GPU.

Methodology Overview

The DeferredGS method decomposes a scene into editable components by attaching learnable texture attributes and normals to each Gaussian, enhanced through normal field distillation from a Signed Distance Function (SDF). For rendering, DeferredGS employs deferred shading, separating the rendering process into two stages: an initial rasterization of geometry and texture attributes into buffer maps followed by pixel-level shading calculations. This two-step process allows for more accurate editing and relighting by addressing the blending issues present in forward shading techniques.

Evaluation and Results

The DeferredGS approach was evaluated against state-of-the-art methods in terms of novel view synthesis, decomposition quality, and editing capability, including texture and geometry editing as well as scene relighting. The quantitative and qualitative results confirm that DeferredGS achieves superior performance in these aspects. Notably, the method standout in its ability to enable real-time editing operations without compromising rendering quality.

Implications and Future Directions

DeferredGS represents a significant advancement in the editing and rendering of 3D scenes, offering both practical and theoretical implications for future research. Practically, it enables more efficient workflows for 3D content creation by allowing for direct edits to texture and geometry with real-time feedback. Theoretically, the work opens avenues for exploring further improvements in rendering techniques and the application of deferred shading in other contexts. Looking ahead, addressing limitations such as shadow baking in textures and noise in edited textures could further enhance this promising avenue of research.

Conclusion

The DeferredGS approach marks a pivotal step towards efficient and editable 3D scene reconstruction, leveraging the strengths of Gaussian Splatting while addressing its limitations in editing capabilities. Through innovative use of deferred shading and normal field distillation, this work lays the groundwork for future advancements in 3D scene editing and rendering technologies.