ForestSplats: Deformable transient field for Gaussian Splatting in the Wild (2503.06179v2)

Abstract: Recently, 3D Gaussian Splatting (3D-GS) has emerged, showing real-time rendering speeds and high-quality results in static scenes. Although 3D-GS shows effectiveness in static scenes, their performance significantly degrades in real-world environments due to transient objects, lighting variations, and diverse levels of occlusion. To tackle this, existing methods estimate occluders or transient elements by leveraging pre-trained models or integrating additional transient field pipelines. However, these methods still suffer from two defects: 1) Using semantic features from the Vision Foundation model (VFM) causes additional computational costs. 2) The transient field requires significant memory to handle transient elements with per-view Gaussians and struggles to define clear boundaries for occluders, solely relying on photometric errors. To address these problems, we propose ForestSplats, a novel approach that leverages the deformable transient field and a superpixel-aware mask to efficiently represent transient elements in the 2D scene across unconstrained image collections and effectively decompose static scenes from transient distractors without VFM. We designed the transient field to be deformable, capturing per-view transient elements. Furthermore, we introduce a superpixel-aware mask that clearly defines the boundaries of occluders by considering photometric errors and superpixels. Additionally, we propose uncertainty-aware densification to avoid generating Gaussians within the boundaries of occluders during densification. Through extensive experiments across several benchmark datasets, we demonstrate that ForestSplats outperforms existing methods without VFM and shows significant memory efficiency in representing transient elements.

ForestSplats: A Novel Approach for Transient Element Representation in 3D Gaussian Splatting

The paper "ForestSplats: Deformable Transient Field for Gaussian Splatting in the Wild" proposes an innovative approach to address the challenges of 3D Gaussian Splatting in real-world scenarios characterized by transient objects and environmental variations. This domain has witnessed significant advancements in recent years, particularly stemming from the efficiency of 3D Gaussian Splatting (3D-GS) in rendering static scenes. However, existing methods fall short when applied to dynamic environments due to prohibitive computational costs and memory inefficiencies.

The authors introduced ForestSplats, leveraging both a deformable transient field and a superpixel-aware mask to explicitly delineate transient elements from static elements in 2D scenes. The proposed method eschews reliance on Vision Foundation Models (VFM), often associated with excessive memory use due to semantic feature extraction. ForestSplats ameliorates this by introducing deformable transient fields capturing transient elements on a per-view basis, enhancing memory efficiency for analysis across unconstrained photographic datasets.

The superpixel-aware mask addresses the need for clear boundary definitions around transient elements. By integrating photometric errors with superpixel segmentation, the approach results in a refined mask that distinguishes static from transient components. This technique is further bolstered by a multi-stage training scheme, which initially optimizes the static field representation before jointly training both static and transient fields. This staged approach reduces masking ambiguity and leads to improved synthesis fidelity.

Moreover, the paper introduces uncertainty-aware densification. Unlike traditional adaptive density control (ADC), which involves random Gaussian sampling, this densification strategy augments positional gradients and pixel coverage computations. This prevents static Gaussians from occupying boundary spaces of transient objects, thereby upholding rendering quality and scene consistency.

The authors present extensive experiments across benchmarks like the NeRF On-the-go and Photo Tourism datasets. Results indicate that ForestSplats reliably decomposes transient elements without VFM utilization, achieving competitive performance against existing state-of-the-art techniques. In particular, tests show memory efficiency improvements—markedly reducing computational overhead compared to methods dependent on pre-computed semantic data.

Implications of this work are multifold, notably in applications requiring real-time rendering from complex visual inputs, such as augmented reality and 3D content generation. The proposal is well-poised to influence future research on transient element management in dynamic 3D scenes. Further exploration may focus on expanding scalable implementations and refining transient field deformability to enhance robustness against diverse occlusion scenarios.

In conclusion, ForestSplats establishes a trail towards efficient and effective transient scene representation, harnessing deformable fields and superpixel segmentation to navigate the challenges posed by transient distractors. It is a significant contribution to the field, showcasing practical and theoretical advancements in the synthesis of novel views from perplexing real-world image collections.