Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections (2403.15704v2)

Abstract: Novel view synthesis from unconstrained in-the-wild images remains a meaningful but challenging task. The photometric variation and transient occluders in those unconstrained images make it difficult to reconstruct the original scene accurately. Previous approaches tackle the problem by introducing a global appearance feature in Neural Radiance Fields (NeRF). However, in the real world, the unique appearance of each tiny point in a scene is determined by its independent intrinsic material attributes and the varying environmental impacts it receives. Inspired by this fact, we propose Gaussian in the wild (GS-W), a method that uses 3D Gaussian points to reconstruct the scene and introduces separated intrinsic and dynamic appearance feature for each point, capturing the unchanged scene appearance along with dynamic variation like illumination and weather. Additionally, an adaptive sampling strategy is presented to allow each Gaussian point to focus on the local and detailed information more effectively. We also reduce the impact of transient occluders using a 2D visibility map. More experiments have demonstrated better reconstruction quality and details of GS-W compared to NeRF-based methods, with a faster rendering speed. Video results and code are available at https://eastbeanzhang.github.io/GS-W/.

• The paper introduces GS-W, which uses 3D Gaussian points to separate intrinsic and dynamic appearance features for improved scene reconstruction.
• It employs adaptive sampling and a 2D visibility map to mitigate transient occluders and manage high-frequency appearance changes.
• Experiments demonstrate superior reconstruction quality and a rendering speed increase of up to 1000x over existing methods.

3D Gaussian Splatting for Novel View Synthesis in Unconstrained Image Collections

Introduction

The endeavor towards novel view synthesis from unconstrained images has seen substantial advancements with the introduction of Neural Radiance Fields (NeRF) and its derivatives. Despite these strides, challenges persist in dealing with photometric variations and transient occluders. In response, we introduce Gaussian in the Wild (GS-W), a paradigm that leverages 3D Gaussian points for scene reconstruction and facilitates distinct intrinsic and dynamic appearance features for each point. This approach not only enhances appearance modeling in varying conditions but also significantly boosts rendering speed.

3D Representations and Novel View Synthesis

Implicit and explicit 3D representations form the foundation for scene reconstruction and 3D object generation. Notably, Neural Radiance Field (NeRF) and its extended methodologies have dominated the landscape, presenting notable successes in synthesizing photorealistic images. However, these techniques typically struggle with non-static scenes encompassing dynamic appearance variations and transient occluders. To address these shortcomings, GS-W incorporates adaptive sampling and a 2D visibility map, significantly mitigating the influence of transient objects and better accounting for high-frequency appearance changes at a local scale.

GS-W Methodology

GS-W proposes a nuanced approach to 3D scene reconstruction using 3D Gaussian points, each endowed with separate intrinsic and dynamic appearance features. This distinction enables a more accurate and flexible representation of the scene under various environmental conditions. Key to this methodology is the introduction of an adaptive sampling strategy, allowing for localized and detailed dynamic appearance modeling. Furthermore, the method leverages a 2D visibility map to minimize the effects of transient occluders, thereby enhancing reconstruction quality and detail.

Experimental Demonstrations and Results

GS-W's performance was rigorously evaluated against existing state-of-the-art methods across multiple datasets. The empirical findings underscore GS-W's superiority in both reconstruction quality and rendering speed, achieving a remarkable 1000x increase in rendering efficiency. These experiments confirm the benefits of separating intrinsic and dynamic appearance features and the practical utility of the adaptive sampling and visibility map strategies in managing the complexities of unconstrained image collections.

Implications and Future Directions

The introduction of GS-W represents a significant advancement in the domain of novel view synthesis, particularly for unconstrained image collections. By addressing the critical challenges associated with dynamic appearance variations and transient occluders, GS-W sets a new benchmark for the quality and efficiency of scene reconstruction. Looking ahead, the framework opens up avenues for further exploration and refinement, particularly in enhancing the model's capability to handle complex lighting variations and specular reflections. Additionally, expanding GS-W's applicability to broader contexts and exploring its potential in related computational photography and vision tasks present exciting prospects for future research.

Conclusion

GS-W heralds a new era in 3D scene reconstruction from unconstrained image collections. Through innovative adaptations such as 3D Gaussian splatting, segregated intrinsic and dynamic appearance features, adaptive sampling, and a visibility map for transient occluder handling, GS-W not only significantly outperforms existing methods in rendering quality and speed but also offers a robust framework for addressing the inherent challenges of working with images captured in dynamic and uncontrolled environments. Moving forward, the continued development and application of GS-W promise to drive further innovations in the field of computer vision and beyond.