Semantic-Guided Progressive Object Removal with Gaussian Splatting

Abstract: Removing unwanted objects from reconstructed 3D scenes is an important task in computer vision, supporting applications in AR/VR, robotics, and digital content creation. Existing methods typically complete the entire masked region in a single step and without effectively utilizing semantic information from other views, leading to difficulties in handling complex geometric details and textures. In this work, we propose a novel framework that integrates Semantic-guided Block Matching (SBM) and Region-Wise Progressive Refinement (RPR) for high-quality 3D object removal. First, we leverage DINOv2 to encode semantic guidance from multi-view observations, and the best match tokens are decoded to complete missing regions in the target view while maintaining cross-view consistency. Second, we introduce a RPR strategy that segments the target mask into multiple subregions and selectively refines those with poor visual quality. Our method is built upon Gaussian Splatting, ensuring high-fidelity scene reconstruction with efficient computation. Experimental results demonstrate that our approach outperforms existing Gaussian-based methods in terms of perceptual quality and coherence in 3D object removal.

• The paper introduces a semantic-guided progressive removal method that harnesses Gaussian Splatting for precise 3D inpainting and object elimination.
• It employs Semantic-Guided Block Matching and Region-Wise Progressive Refinement to integrate multi-view semantic cues and enhance high-frequency details.
• Experimental results demonstrate significant improvements in PSNR, SSIM, and computational efficiency, advancing 3D scene editing for AR/VR and robotics.

Semantic-Guided Progressive 3D Object Removal with Gaussian Splatting

Introduction

The paper "Semantic-Guided Progressive Object Removal with Gaussian Splatting" (2607.04144) introduces a framework addressing the problem of high-fidelity object removal in reconstructed 3D scenes, notably within the context of Gaussian Splatting (GS) representations. This task is vital for AR/VR, robotics, and general digital scene manipulation, and is characterized by the need for visually and geometrically consistent view synthesis after object removal, especially in complex or unbounded environments. The framework distinguishes itself by integrating multi-view semantic guidance and a progressive refinement strategy based on high-frequency detail supervision, explicitly addressing persistent limitations in state-of-the-art GS- and NeRF-based pipelines.

Methodology

Overview

The proposed technique combines three major components:

• Semantic-Guided Block Matching (SBM): Utilizes multi-view feature encoding and cross-view semantic alignment.
• Region-Wise Progressive Refinement (RPR): Identifies and re-inpaints low-detail regions progressively.
• High-Frequency Feature Extraction: Enforces sharpness and preserves high-frequency details.

All steps are embedded in a GS pipeline, ensuring efficient, real-time capable, and photorealistic 3D object removal. The pipeline first removes targeted object primitives using 2D masks, initializes Gaussians in mask regions, and bootstraps inpainting with Score Distillation Sampling (SDS) using a diffusion model backbone, then applies semantic guidance and localized iterative refinement.

Figure 2: Given a 3D scene, the method produces accurate object removal with region refinement and outputs a visually consistent 3D scene.

Semantic-Guided Block Matching

Initial inpainting is performed, and the masked region is decomposed into small square blocks. For each block, a DINOv2 encoder extracts semantic embeddings, which are used to retrieve the most semantically similar patches from other views using cosine similarity. The best match token conditions a diffusion-based inpainting process via cross-attention in the UNet, replacing the conventional text embedding. This design leverages explicit cross-view semantic information far beyond existing one-shot or 2D-centric inpainting methods, directly promoting consistency and fidelity across novel views.

High-Frequency Feature Supervision

Post-inpainting, image gradients (Sobel filters) in the reconstructed RGB images are used to detect poorly detailed regions. Low-frequency (i.e., blurry or over-smoothed) blocks are flagged for selective refinement. A frequency-aware embedding is computed and concatenated with the semantic token as additional conditioning for the diffusion-based re-inpainting step. By focusing computational effort on blocks lacking perceptual detail, sharpness and local realism are enhanced while computational cost remains tractable.

Region-Wise Progressive Refinement (RPR)

The RPR module conducts iterative block-level re-inpainting. Only blocks with insufficient high-frequency response are refined, and this process is repeated until all masked regions achieve an acceptable perceptual/thr aesthetic threshold. This targeted and progressive approach mitigates the problem of over-smoothed or geometrically inconsistent fills typical in prior diffusion-based 3D inpainting methods.

Integration with GS

The entire mechanism is encapsulated within a 3D Gaussian Splatting framework, with adaptive densification, coarse-to-fine optimization, and efficient parameterization. Loss functions include standard reconstruction and differentiable SSIM for GS, plus SDS for diffusion prior alignment, with semantic and frequency-aware terms weighted for optimal view and perceptual consistency.

Experimental Results

Empirical evaluation is conducted on public and self-captured datasets spanning forward-facing to 360-degree scenes, benchmarking against state-of-the-art NeRF- and GS-based object removal methods (e.g., SPIn-NeRF, GaussianEdit, MVInpainter, InFusion).

Qualitative Comparison

Figure 1: Qualitative results comparing the proposed method, SPIn-NeRF, and GaussianEdit in a consistent view. The method achieves visibly superior view-consistent inpainting.

Figure 3: Qualitative results on Bear and Bicycle scenes, visualizing the removal results of MVInpainter, GaussianGroup, InFusion, and the proposed method; ours yields sharper, more coherent completions.

Across all scenes, the method produces sharper textures and more geometrically plausible scene completions. Competing approaches frequently exhibit ghosting, texture discontinuities, or over-smoothing, particularly when faced with complex geometric occlusions or cluttered backgrounds.

Quantitative Metrics

• The method establishes the highest reported PSNRs (e.g., 28.7 on Rednet, 29.6 on Chuyin), outperforming prior GS-based pipelines by significant margins.
• Corresponding SSIM and LPIPS values confirm advances in both view-consistent structural reconstruction and perceptual similarity.
• Ablation reveals that SBM induces a PSNR gain of 5.57–4.44 over variants lacking semantic guidance, while RPR provides notable improvement over one-shot approaches.

Efficiency

Training and inference runtimes, as well as GPU memory usage, are favorable relative to prior GS-diffusion methods. The selective block refinement streamlines computation, supporting practical deployment scenarios.

Implications and Future Directions

Practical Implications

Integrating explicit multi-view semantics with progressive, frequency-aware block-level refinement within a GS-optimized pipeline sets a new standard for editable 3D scene reconstruction. This framework directly enables improved authoring, AR/VR scene preparation, and robot perception pipelines that rely on dynamic, editable 3D assets. The efficient use of multi-view semantic cues—rather than generic or single-view inpainting—could be harnessed for robust downstream scene understanding and novel geometry generation tasks.

Theoretical Insights

The explicit block-wise semantic alignment, coupled with localized frequency-based perceptual supervision, highlights the need for spatially and semantically aware diffusion conditioning in 3D generative pipelines beyond pixel-level metrics. The paradigm shift toward hybrid global-local, semantic-frequency guidance may inspire future cross-view self-supervised 3D models and more advanced GS-oriented architectures with built-in semantic and frequency priors.

Future Developments

The proposed technique could be synergistically expanded with large-scale pretraining of multi-view diffusion models, extension to strongly dynamic or deformable scenes, and fusion with non-Gaussian parametric 3D representations. Additionally, the block-wise progressive refinement paradigm could be applied to other scene-editing tasks (insertion, semantic relabeling, compositional editing), especially in zero-shot or generalizable 3D learning settings.

Conclusion

This work presents a semantically guided, progressively refined framework for 3D object removal within Gaussian Splatting, characterized by explicit cross-view semantic matching and frequency-based selective refinement. Significant improvements are demonstrated on standard benchmarks, with strong numerical results and advances in perceptual and geometric consistency. The approach points to a promising trajectory for editable, efficient, and high-fidelity scene manipulation in next-generation vision and robotics pipelines.