---
title: 'SplatFill: 3D Inpainting for 3D Gaussian Splatting'
url: https://www.emergentmind.com/topics/splatfill
type: topic
---

# SplatFill: 3D Inpainting for 3D Gaussian Splatting

SplatFill is a 3D scene inpainting method for 3D Gaussian Splatting (3DGS) that targets object removal, occlusion completion, and geometry completion required for consistent novel-view rendering. It takes multi-view RGB images, camera poses, and binary masks indicating regions that should be removed and then plausibly filled, and it produces a complete editable 3DGS scene representation rather than a collection of independently edited 2D images [2509.07809]. Its central design combines a single high-quality 2D inpainted reference view, monocular depth priors, object-aware supervision, and an iterative consistency-aware refinement procedure called Selective Guided Inpainting, or SGI, to improve geometric placement, visual coherence, and efficiency in masked regions [2509.07809].

## 1. Problem setting and motivation

SplatFill addresses **3D scene inpainting** in a multi-view 3DGS setting where missing regions arise from occlusion or scene editing. Typical examples include object removal from a reconstructed scene, filling content revealed after editing, recovering surfaces that were previously occluded, and completing geometry needed for consistent novel-view rendering [2509.07809]. The output must remain plausible under viewpoint change, which makes the task materially different from standard 2D image inpainting.

The method is motivated by three recurring failure modes in prior pipelines. First, independent 2D inpainting across views often yields slight cross-view discrepancies that become ghosting, geometry conflicts, blurry splats, and instability after 3D optimization. Second, plausible 2D completions do not by themselves determine correct 3D placement, which can lead to floating Gaussians, incorrect depths, broken boundaries, and inconsistent geometry. Third, prior NeRF-based and 3DGS-based inpainting systems often produce oversmoothed textures, blurred details, and artifacts, especially for viewpoints far from a single reference image [2509.07809].

Within the paper’s taxonomy of prior work, NeRF-based methods such as SPIn-NeRF, OR-NeRF, MVIP-NeRF, Inpaint3D, NeRFiller, and reference-guided NeRF inpainting address parts of the problem but incur trade-offs in speed, detail, or multi-view consistency. Prior 3DGS methods are described as falling into two common paradigms: single-reference conditioning, which can fail under large viewpoint changes, and multi-view aggregation of inpainted images, which can blur or artifact when the 2D generations are not fully consistent. SplatFill is formulated as a response to those limitations by combining sharp-detail initialization from one inpainted view with multi-view geometric guidance, object-aware supervision, and selective iterative refinement rather than global re-inpainting [2509.07809].

## 2. Pipeline and system components

The pipeline is explicitly organized into seven stages. It first selects a reference view and performs 2D inpainting of its masked region. It then estimates monocular depth maps for the inpainted reference and the other masked views, groups pixels by depth level to obtain reliable depth priors, applies instance segmentation on training images, initializes and optimizes Gaussians in missing regions using the inpainted reference image together with depth and object-aware constraints, detects inconsistent views by comparing monocular depth and rendered depth, and then repeats refinement until the views satisfy geometric and visual consistency requirements [2509.07809].

Several external modules are part of the system definition. The reference-view completion is produced by a diffusion-based 2D inpainting method from Esser et al. (2024). Monocular depth is obtained with **Depth Anything**, and instance segmentation masks are obtained using **SAM2**. The 3DGS optimizer is implemented on a modified **MCMC** framework whose rasterizer is extended to render depth [2509.07809].

The method is not a new renderer from scratch. It builds on standard 3DGS and retains explicit Gaussian scene representation, differentiable splat rasterization, alpha compositing, and optimization of geometry and appearance parameters. Its novelty lies in inpainting-specific supervision and refinement. The initial missing content is introduced through one selected reference image, whose masked area is filled in 2D and then used as the first direct supervision signal for the hidden region. The paper is explicit that it does not provide a separate closed-form formula for Gaussian initialization in the masked region; instead, new or updated Gaussians in missing areas are placed and refined through optimization under photometric, depth, and object-aware signals [2509.07809].

An initial baseline reconstruction is established with **8000 steps** before selective refinement begins. This division between baseline optimization and later targeted correction is central to the method’s efficiency claim, because only inconsistent views and subregions are revisited during SGI [2509.07809].

## 3. Depth-guided and object-aware supervision

Depth supervision is one of the two principal supervision channels. The paper argues that a 2D inpainted image provides appearance but not necessarily correct 3D placement, so monocular depth priors are used to place inpainted content in plausible depth ranges, reduce floating or outlier Gaussians, improve volumetric consistency across views, and identify inconsistent regions during refinement [2509.07809].

The core depth term is the **Soft Depth Clustering Loss** (SDCL). Instead of direct metric depth supervision, which would require scale-and-shift alignment between monocular depth and rendered depth, SDCL partitions the estimated depth map into evenly spaced depth bins, constructs a mask $\mathcal{M}_k$ for each bin, computes the mean rendered depth $\mu_k$ inside that bin, and penalizes deviations of rendered depths from the within-bin mean. This avoids explicit metric alignment and weights distant bins less heavily because monocular depth is less reliable there [2509.07809]. The paper presents this as a key novelty and argues that it preserves finer details for far objects better than conventional metric depth supervision.

A second depth term, the **Crop-Focused Depth Loss** (CFDL), concentrates supervision near the inpainted area. Every **9** iterations, a crop is taken around the inpainted region, randomly expanded to include context, processed with the same binning strategy as SDCL, and weighted by a Gaussian-like function that emphasizes pixels near the center of the inpainted area. The total depth objective is the sum of SDCL and CFDL with $\kappa = 25$ [2509.07809].

The second supervision channel is object-aware. Each Gaussian is augmented with a **16-dimensional learnable feature vector**, and rendered feature maps are grouped by **SAM2** instance masks. The resulting **Object-Aware Contrastive Loss** (OACL) encourages features from the same segment to cluster and features from different segments to separate [2509.07809]. In the paper’s framing, depth indicates where content should be, while object priors indicate what belongs together structurally. This dual supervision is intended not only to improve reconstruction quality but also to support later tasks such as 3D object selection [2509.07809].

A common misconception is that SplatFill relies only on depth. The paper’s design is explicitly bimodal: depth regularizes spatial placement, while object supervision regularizes structural coherence and object boundaries. The authors also note a limitation here: segmentation groups Gaussians by object, but the method does not yet segment and complete each 3D object independently [2509.07809].

## 4. Consistency-aware refinement with Selective Guided Inpainting

The refinement stage, **Selective Guided Inpainting** (SGI), addresses the failure case in which a model optimized from a single inpainted reference remains inconsistent in views that are far from that reference. Rather than globally re-inpainting all masked content, SGI renders all training views from the current 3DGS model, compares rendered depth with monocular depth inside masked regions, and identifies the view with the largest cumulative inconsistency for targeted correction [2509.07809].

The refinement signal is computed from the per-pixel absolute depth discrepancy
$$
E(x,y) = \left| D_{\text{rend}(x,y) - D_{\text{mono}(x,y) \right|.
$$
For each view, the error is summed over masked-region pixels, and the view with the largest cumulative error is selected as $v^*$. Within that view, the gradient magnitude
$$
G(x,y) = \left\| \nabla E(x,y) \right\|_2
$$
is used to localize sharp inconsistency boundaries such as misplaced surfaces and depth discontinuities [2509.07809]. The implementation uses the Sobel operator, erodes the base mask to reduce boundary artifacts, thresholds the gradient map, and applies morphological dilation to obtain a refined binary mask $B(x,y)$ that marks only the problematic subregion.

Only that selected subregion is re-inpainted in 2D. The updated image is then added to the reference set, and 3DGS optimization resumes. This process repeats until the discrepancies become small or geometric and visual consistency is reached [2509.07809]. The paper’s rationale is that global re-inpainting is both expensive and destabilizing: it may overwrite regions that are already correct and can inject inconsistent 2D generations into the 3D model. SGI preserves already coherent regions and concentrates computation on the largest residual errors.

This refinement policy is also one of the method’s primary efficiency mechanisms. The abstract attributes the training-time gain to a consistency-aware refinement scheme that selectively identifies and corrects inconsistent regions without disrupting the rest of the scene, and the reported reduction relative to GScream is **24.5%** [2509.07809].

## 5. Mathematical formulation and representation

SplatFill retains the standard 3DGS alpha-compositing renderer. For a pixel $p$, the rendered color is
$$
C(p) = \sum_{i=1}^{|\mathcal{G}_p|} \mathbf{c}_i\, \alpha_i' \prod_{j=1}^{i-1} \left(1 - \alpha_j'\right),
$$
where $\mathcal{G}_p$ is the set of Gaussians affecting $p$, $\mathbf{c}_i$ is the color of Gaussian $i$, and $\alpha_i'$ is the effective opacity contribution of that Gaussian at the pixel [2509.07809]. Each Gaussian includes 3D position $\boldsymbol{\mu}$, covariance $\Sigma$, color $\mathbf{c}$, opacity $\alpha$, spherical harmonics coefficients, and the additional 16-dimensional feature vector used for OACL [2509.07809].

The per-bin SDCL term is
$$
L_k = \frac{1}{|\mathcal{M}_k|} \sum_{p \in \mathcal{M}_k} \left| d(p) - \mu_k \right|,
$$
and the weighted global SDCL is
$$
\mathcal{L}_{\text{SDCL} = \frac{\sum_{k} w_k \cdot L_k}{\sum_{k} w_k}
= \frac{\sum_{k} \frac{w_k}{|\mathcal{M}_k|}\sum_{p\in \mathcal{M}_k} \left| d(p) - \mu_k \right|}{\sum_{k} w_k}.
$$
The full depth term is
$$
\mathcal{L}_{\text{depth} = \mathcal{L}_{\text{SDCL} + \kappa \, \mathcal{L}_{\text{CFDL},
$$
with $\kappa = 25$ [2509.07809].

The object-aware feature loss is
$$
\mathcal{L}_{\text{OACL} = -\frac{1}{N_k} \sum_{p=1}^{N_k} \frac{1}{|\{f_p\}|} \sum_{q=1}^{|\{f_p\}|} \log \frac{ \exp\left( \frac{f_p^q \cdot \bar{f}_p}{\phi_p} \right) }{ \sum_{s=1}^{N_k} \exp\left( \frac{f_p^q \cdot \bar{f}_s}{\phi_s} \right) },
$$
with temperature
$$
\phi_p = \frac{1}{N_p} \sum_{q=1}^{N_p} \|f_p^q - \bar{f}_p\|_2 \cdot \log (N_p + \epsilon),
$$
where $\epsilon = 100$ [2509.07809]. All features are $\ell_2$-normalized before loss computation.

The paper repeatedly refers to optimization of photometric parameters and supervision by the images, but it does **not** print a single full total-loss formula combining photometric, depth, and OACL terms in the provided text [2509.07809]. That omission matters for reproduction, because the paper’s novel losses are specified precisely while the exact global weighting of all objectives is not.

## 6. Evaluation, comparative performance, and empirical findings

The main benchmark is **SPIn-NeRF**, which contains **10 scenes** and **100 multi-view images per scene** with human-annotated object masks. For each scene, **60 training images** contain the unwanted object and **40 test images** show the scene without the object [2509.07809]. Evaluation uses **PSNR**, **SSIM**, **LPIPS**, and **FID**, computed both over the full image and over the masked region only [2509.07809].

On full-image evaluation, SplatFill reports **PSNR 20.46**, **SSIM 0.63**, **LPIPS 0.25**, **FID 29.76**, and **34 min** training time. The corresponding numbers reported for GScream are **20.45**, **0.58**, **0.28**, **36.72**, and **45 min**; SPIn-NeRF reports **20.18**, **0.46**, **0.47**, **58.78**, and approximately **112.5 min**; OR-NeRF reports **20.32**, **0.54**, **0.35**, **38.69**, and approximately **225 min** [2509.07809]. The abstract summarizes these results as state-of-the-art perceptual quality and improved efficiency, with the training-time reduction relative to GScream quantified as **24.5%** [2509.07809].

On masked-region evaluation, SplatFill reports **PSNR 15.67**, **SSIM 0.21**, and **LPIPS 0.54**, essentially matching GScream on those three metrics while remaining stronger on the full-image evaluation [2509.07809]. The paper interprets this as evidence that SplatFill preserves or improves overall scene coherence without sacrificing masked-region competitiveness.

The ablations isolate the method’s main components. For depth-based supervision, the masked-region numbers are **15.14 / 0.18 / 0.56** without depth supervision, **15.22 / 0.19 / 0.56** with the GScream depth loss, and **15.67 / 0.21 / 0.54** with SplatFill’s full depth loss, indicating that **SDCL + CFDL** improves Gaussian placement and perceptual quality [2509.07809]. For the inpainting strategy, **Single Reference** gives **15.35 / 0.19 / 0.54**, while **Selective Guided Inpainting** gives **15.67 / 0.21 / 0.54**, supporting the claim that one reference is insufficient and that selective multi-view correction improves robustness [2509.07809].

Qualitatively, the paper attributes to SplatFill sharper textures, better geometric coherence, fewer artifacts, and stronger consistency in difficult viewpoints. The examples highlighted include sharper edges of a stone bench, finer grass details behind fences, and more detailed grass around a manhole, especially where GScream becomes blurred [2509.07809].

## 7. Position within related research, applications, and limitations

Within the broader splat-based and diffusion-based literature, SplatFill occupies a specific niche: explicit **3D scene inpainting** for 3DGS. This distinguishes it from methods that emphasize other forms of control or lifting. **ControlFill** is a latent-diffusion 2D inpainting method with learned creation and removal conditions and a spatial guidance tensor, but it is explicitly described as **not** a 3D-aware, multi-view consistent, geometry-aware, or Gaussian-splatting method [2503.04268]. **SplatDiff** is a pixel-splatting-guided latent video diffusion model for novel view synthesis that fills disocclusions and reduces texture hallucination, but it is not formulated as explicit 3DGS scene inpainting [2502.12752]. **FluSplat** addresses sparse-view text-guided 3D editing without test-time optimization and emphasizes cross-view consistent editing rather than explicit filling of unseen or occluded regions [2604.20038]. **FSFSplatter** improves sparse-view, pose-free Gaussian-splatting reconstruction through dense initialization, self-splitting densification, and geometry-enhanced optimization, but it is described as adjacent rather than a direct scene-completion method [2510.02691]. **Splat Feature Solver** assigns and denoises semantic features on existing splats without adding geometry [2508.12216], and **Splat-LOAM** performs LiDAR odometry and mapping with local densification over observed surfaces rather than generative scene completion [2503.17491]. This positioning suggests that SplatFill is one of the more explicit treatments of the 3DGS inpainting problem itself.

The method’s practical use cases include object removal, occlusion completion, scene editing, and content insertion. The supplementary material is described as showing that the same pipeline can be used for object insertion, using a text-prompted 2D inpainting result as a reference and propagating it into a coherent 3D scene [2509.07809]. The paper also lists AR/VR, game design, film post-production, and robotics among application domains that benefit from fast editable 3D scenes [2509.07809].

The paper is explicit about several limitations. The initial reference view is selected randomly, and its quality strongly affects reconstruction quality. Object supervision is underused in the sense that the method does not yet segment and complete each 3D object independently. Illumination changes, including shadows and reflections, are not explicitly handled and can still produce inconsistencies across views [2509.07809]. These caveats delimit the scope of the reported gains: SplatFill is strongest when the reference inpainting is high quality, monocular depth is sufficiently informative, and the remaining inconsistencies are localized enough for SGI to correct selectively.

In summary, SplatFill is best characterized as a depth-guided 3DGS inpainting framework that begins from a single high-quality 2D inpainted reference, augments standard Gaussian optimization with SDCL, CFDL, and OACL, and then uses SGI to correct only those views and subregions where rendered geometry still disagrees with monocular depth. Its reported contribution is not a new renderer, but an inpainting-specific supervisory and refinement regime for producing geometrically coherent, sharper, and more efficient 3D scene completions in the 3DGS setting [2509.07809].

Source: https://www.emergentmind.com/topics/splatfill