GA-GS: Generation-Assisted Gaussian Splatting for Static Scene Reconstruction

Abstract: Reconstructing static 3D scene from monocular video with dynamic objects is important for numerous applications such as virtual reality and autonomous driving. Current approaches typically rely on background for static scene reconstruction, limiting the ability to recover regions occluded by dynamic objects. In this paper, we propose GA-GS, a Generation-Assisted Gaussian Splatting method for Static Scene Reconstruction. The key innovation of our work lies in leveraging generation to assist in reconstructing occluded regions. We employ a motion-aware module to segment and remove dynamic regions, and thenuse a diffusion model to inpaint the occluded areas, providing pseudo-ground-truth supervision. To balance contributions from real background and generated region, we introduce a learnable authenticity scalar for each Gaussian primitive, which dynamically modulates opacity during splatting for authenticity-aware rendering and supervision. Since no existing dataset provides ground-truth static scene of video with dynamic objects, we construct a dataset named Trajectory-Match, using a fixed-path robot to record each scene with/without dynamic objects, enabling quantitative evaluation in reconstruction of occluded regions. Extensive experiments on both the DAVIS and our dataset show that GA-GS achieves state-of-the-art performance in static scene reconstruction, especially in challenging scenarios with large-scale, persistent occlusions.

• The paper presents a novel GA-GS framework that integrates geometric priors and diffusion inpainting to overcome occlusion challenges in static scene reconstruction.
• It leverages authenticity-aware opacity modulation to blend real and generated data, achieving state-of-the-art PSNR and SSIM on challenging benchmarks.
• The method demonstrates practical benefits for VR, AR, and robotics by effectively recovering clutter-free static backgrounds from dynamic video inputs.

GA-GS: Generation-Assisted Gaussian Splatting for Static Scene Reconstruction

Introduction

This work introduces GA-GS, a generation-assisted 3D Gaussian Splatting (3DGS) framework targeting the reconstruction of high-fidelity static 3D scenes from monocular videos containing dynamic objects. The method addresses scenarios in which persistent occlusion due to dynamic foregrounds poses significant obstacles to reliable static background recovery. The core contributions are threefold: a pipeline integrating geometric priors from visual foundation models, diffusion-based inpainting for occlusion supervision, and authenticity-aware rendering that balances the trade-off between fidelity and supervision completeness. Extensive evaluation on standard and newly constructed datasets validates the superiority of GA-GS, particularly in the occlusion-dominated regime.

Figure 1: The GA-GS pipeline leverages diffusion inpainting for auxiliary supervision and introduces authenticity-driven rendering to combine real and generated regions.

Background and Motivation

Prior 3DGS-based techniques for static scene reconstruction segment out dynamic objects and supervise only on known-background regions. While effective in controlled cases, these strategies are brittle under circumstances of significant or long-lived foreground occlusion, resulting in unreconstructable areas and visual artifacts. At the opposite end, generative approaches can hallucinate plausible static regions via powerful priors (e.g., inpainting models), but inevitably introduce lower-fidelity, less-trustworthy signals. Thus, the central challenge is to devise a representation and supervision regime that can robustly utilize both observed and generated data while appropriately weighting their contributions as per their trustworthiness.

Methodology

Pipeline Overview

GA-GS comprises four principal modules: camera and geometry estimation, dynamic mask generation, diffusion-based occlusion synthesis, and authenticity-aware blending.

Figure 2: Schematic of the GA-GS pipeline illustrating all modules and information flow.

Geometric Prior Construction

Initial camera poses and dense point clouds are obtained using the VGGT visual geometry transformer, circumventing the need for traditional SfM methods and facilitating application to uncalibrated monocular sequences. Confidence-filtered, structure-aware sampling is applied to retain salient geometric support while suppressing noisy or redundant points. These act as anchors for the 3D Gaussian primitives.

Motion-Aware Masking

Dynamic regions are automatically segmented via Flow-SAM, which combines the generalization strength of the SAM model's segmentation with dense optical flow cues (e.g., via RAFT) for temporally consistent occlusion masks.

Diffusion-Based Supervision Synthesis

Occluded regions, as signaled by binary masks, are inpainted using DiffuEraser. The resulting pseudo-backgrounds are composited with the original frames, yielding per-pixel supervision that is real where available and generative elsewhere. This hybrid supervision boosts the spatial support for learning static content even in heavily occluded scenarios.

Authenticity-Aware Opacity Modulation

A scalar authenticity parameter $\theta \in [0,1]$ is attached to each Gaussian primitive. It is initialized to reflect the provenance of the underlying pixel (e.g., 0.9 for real, 0.1 for inpainted) and is further optimized during training. This parameter modulates the effective opacity of primitives in both rendering and loss computation, controlling the influence of real and generated content adaptively. The supervised loss employs pixel-wise weighting that prioritizes learning from the real background while exploiting the generative regions as auxiliary signals.

Optimization

The pipeline utilizes weighted pixel-wise L1 loss and clamped SSIM loss as objectives. Differentiable rasterization renders images from the current set of Gaussians, which are subject to densification and pruning (per 3DGS tradition). The model is trained with fixed, pretrained external modules to ensure fairness in ablation and baseline comparisons.

Experimental Validation

Datasets

Evaluation is conducted both on DAVIS, focusing on challenging dynamic sequences, and the new Trajectory-Match dataset. The latter contains pairs of matched videos: one with dynamically moving objects and another with the static background captured along the same camera trajectory, facilitating direct, artifact-free ground-truth evaluation even in previously occluded regions.

Figure 3: The Trajectory-Match dataset uses a robot-mounted camera to capture precisely aligned dynamic and static scene sequences for rigorous benchmarking.

Results

GA-GS achieves new state-of-the-art performance for static scene reconstruction on both benchmarks. On DAVIS, PSNR improvements are observed across all challenging scenes compared to competing baselines such as WildGaussians, Robust3DGS, and DAS3R. On Trajectory-Match, strong gains are maintained for all metrics (PSNR/SSIM/LPIPS) at both scene- and occlusion-level evaluations.

Qualitative visualizations further substantiate the effectiveness of generation-assisted supervision and authenticity-aware modulation in reconstructing background content and filling large occluded regions with structurally coherent geometry and appearance.

Figure 4: On DAVIS, GA-GS yields superior occlusion removal and background recovery relative to prior methods.

Figure 5: Visualizations on Trajectory-Match—ground-truth, GA-GS, and baseline reconstructions demonstrate GA-GS’s capacity for high-fidelity, artifact-free recovery even under severe dynamic occlusion.

Ablations

Ablation experiments establish:

• The removal of the authenticity parameter or the generative module substantially degrades performance, particularly in occluded areas.
• The initialization of $\theta$ critically influences results, with best performance achieved when assigning real and inpainted content distinct, appropriately calibrated priors (e.g., $(0.9, 0.1)$).
• The pipeline’s gains are not contingent on pose estimation source (VGGT vs. COLMAP), indicating method-agnosticity in geometric prior acquisition.

Implications and Future Directions

Practically, GA-GS enables robust scene digitization for VR, AR, robotics, and autonomous driving, with resilience against transient or persistent scene clutter and occlusion. The combination of generative models with authenticity-modulated blending offers a generalizable template for “trust-aware” use of synthetic data in hybrid real+generated supervision. The release of Trajectory-Match enables the community to further explore metrics and architectures for occluded-region scene completion.

Theoretically, GA-GS’s authenticity calibration introduces a learnable mechanism for managing supervision-source uncertainty—a concept likely extensible to other modalities and generative-complemented tasks. Exciting future work includes the integration of stronger semantic priors, continuous uncertainty estimation, and active view planning to maximize static scene coverage in ambiguous or underconstrained environments.

Conclusion

GA-GS advances static scene reconstruction under dynamic occlusion by synergizing foundation model-based geometry, motion segmentation, diffusion inpainting, and authenticity-aware rendering. It demonstrates quantitatively and qualitatively superior reconstructions, validated on both standard and purpose-built datasets, and sets new benchmarks for artifact-minimized, high-fidelity scene recovery in realistic, unconstrained videos.