GS-RoadPatching: 3DGS Inpainting for Driving Scenes

• The paper introduces a substitutional inpainting pipeline that fills missing regions in 3DGS driving scenes by reusing existing Gaussian patches.
• It leverages feature-embedded neural Gaussians and a manifold bird’s-eye-view search for efficient, structure-aware patch matching.
• Experimental results demonstrate improved LPIPS, FID, and render quality, supporting dynamic-object removal and enhanced downstream editability.

GS-RoadPatching is a 3D Gaussian Splatting (3DGS)-native inpainting method for driving scene completion that fills missing or edited regions by searching for compatible content in already reconstructed parts of the same 3D scene and then substituting that content directly in Gaussian space, rather than generating 2D inpainting priors and retraining the scene. In the formulation of "GS-RoadPatching: Inpainting Gaussians via 3D Searching and Placing for Driving Scenes," the method targets dynamic-object removal and scene completion under novel-view rendering, temporal coherence, and downstream editability, and it is explicitly framed as a substitutional inpainting pipeline built on feature-embedded neural Gaussians, patch-anchor indexing, manifold bird’s-eye-view (BEV) search, and reprojection-guided fusion (Chen et al., 24 Sep 2025).

1. Problem formulation and conceptual framing

GS-RoadPatching addresses scene completion for reconstructed driving scenes in which target regions correspond to moving objects, incompletely reconstructed areas, or manually selected edit regions. The input is a driving video sequence reconstructed as a 3DGS-style scene; the output is a completed 3D Gaussian scene that remains usable for novel-view rendering and further 3D editing. The paper emphasizes that direct inpainting in 3DGS is difficult because 3D patch correspondence is ambiguous, standard Gaussian attributes are not optimized for structural comparison between local 3D regions, and the result must remain valid under multi-view rendering rather than only from a single perspective (Chen et al., 24 Sep 2025).

The central conceptual distinction is between generative completion and substitutional inpainting. In generative completion, missing content is hallucinated from learned priors such as diffusion, GAN, or view-conditioned depth/RGB prediction. In GS-RoadPatching, the missing region is completed by reusing existing Gaussians from another region of the same scene. The paper presents this as more interpretable, more controllable, less dependent on cross-view consistency of 2D priors, and cheaper because it avoids time-intensive Gaussian retraining (Chen et al., 24 Sep 2025).

A recurring misconception is to treat GS-RoadPatching as a 2D video inpainting method wrapped around 3DGS. The method is defined in the opposite direction: it operates directly on the reconstructed 3DGS scene and uses 3D searching and placing as the primary completion mechanism. This places it in contrast with 2D-prior-guided pipelines such as InFusion, RefFusion, and related methods, which first generate inpainted RGB or depth priors and then re-optimize a Gaussian scene (Chen et al., 24 Sep 2025).

2. Feature-embedded neural Gaussian representation

The method builds on an anchor-based neural Gaussian representation associated with Scaffold-GS and uses Neural Scene Graphs for dynamic scenes. Scene construction uses SfM and LiDAR-stitched point clouds for initialization, then jointly optimizes radiance, depth consistency, and feature consistency. The reconstructed neural Gaussians carry color, rotation, scale, opacity, anchor feature, offset, and an additional visual feature embedding used for patch matching. The appendix specifies that the default embedding is a 256-dimensional SuperPoint feature (Chen et al., 24 Sep 2025).

Feature supervision is explicit. For a training image $I \in \mathbb{R}^{H \times W \times 3}$, the extracted ground-truth feature map is denoted $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$, and the rendered feature map from the Gaussian feature field is $\hat{F}_s(I)$. The feature loss is

$$\mathcal{L}_{\mathrm{Feat}} = M_i \cdot \|F_t(I) - \hat{F}_s(I)\|_1,$$

where $M_i$ is a dynamic-object mask. The purpose of $M_i$ is to suppress interference from dynamic foreground objects when learning static-background features. The full reconstruction objective is

$$\mathcal L = (1-\lambda_{\mathrm{SSIM}})\mathcal L_1 + \lambda_{\mathrm{SSIM}}\mathcal L_{\mathrm{SSIM}} + \lambda_{\mathrm{Depth}}\mathcal L_{\mathrm{Depth}} + \lambda_{\mathrm{Feat}}\mathcal{L}_{\mathrm{Feat}},$$

with $\lambda_{\mathrm{SSIM}} = 0.2$, $\lambda_{\mathrm{Depth}} = 0.2$, and $\lambda_{\mathrm{Feat}} = 1.0$ (Chen et al., 24 Sep 2025).

This representation is not introduced merely as a richer renderer. Its specific role is to make local 3D regions comparable. Vanilla Gaussian parameters suffice for image formation, but the paper argues that they are inadequate for robust patch matching. The addition of visual feature embeddings turns the Gaussian scene into a searchable structural database. A plausible implication is that the method depends as much on descriptor quality as on rendering quality, which is consistent with the ablation showing that color-only matching is markedly weaker than learned feature matching (Chen et al., 24 Sep 2025).

3. Patch definition, anchor indexing, and manifold BEV search

A target patch $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$0 is defined as a continuous group of anchors, and a candidate source patch $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$1 is another anchor group from elsewhere in the same scene. Anchors within a target patch are categorized as missing, incomplete, or intact. The paper states that incomplete anchors are not used for contextual source search because they lack reliable supervision. This distinction is important: patch matching is based on intact contextual structure rather than on the missing core itself (Chen et al., 24 Sep 2025).

To make search tractable, the method overlays a voxel grid on the patch using a voxel size parameter $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$2 and builds a hash-encoded bidirectional index between anchors and voxels, as well as between patches and anchor contents. The paper describes this as volumetric patch-anchor indexing. Although no closed-form indexing equation is given, the design is explicit: the index supports efficient retrieval of local Gaussian subsets during search and substitution (Chen et al., 24 Sep 2025).

The actual search is a road-scene-specialized manifold BEV search. Rather than performing an unconstrained 3D nearest-neighbor lookup, the method decomposes the ground manifold into two main directions with respect to road elements and driving direction, constructs a BEV space orthogonal to the manifold surface, and performs bidirectional path search along the $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$3-axis for a target patch. This yields a rectangular search region on the manifold. The underlying assumption is that road scenes exhibit repetitive structure and that plausible source patches are likely to lie on nearby road-surface neighborhoods rather than at arbitrary 3D locations (Chen et al., 24 Sep 2025).

Patch compatibility is measured through feature affinity. For projected target and source supports, $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$4 and $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$5, the affinity is

$F_t(I) \in \mathbb{R}^{H' \times W' \times D}$6

where the bracket denotes cosine similarity between bilinearly interpolated rendered features from viewpoints nearest the target and source. The best source patch is selected by

$F_t(I) \in \mathbb{R}^{H' \times W' \times D}$7

The method therefore uses geometry implicitly through anchor organization and BEV-restricted search, but its explicit similarity score is feature-based rather than a hand-designed geometry term (Chen et al., 24 Sep 2025).

4. Substitution, reprojection-guided fusion, and inpainting-time optimization

Once the best source patch $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$8 is identified, the method performs preliminary inpainting by transferring neural Gaussians in anchor-based units. The paper describes this as voxel-based patch substitution and notes that Gaussians outside the target area are suppressed by setting opacity to zero according to anchor positions and spatial offsets. No explicit rigid alignment transform or ICP-like registration objective is specified; placement is implied by the matched anchor and voxel organization (Chen et al., 24 Sep 2025).

Direct substitution is not considered sufficient for visual harmony. GS-RoadPatching therefore adds reprojection-guided fusion. The motivating observation is that, although a dynamic object may occlude background in some views, the same background may be visible in other frames. The method extracts a clean RGB background patch $F_t(I) \in \mathbb{R}^{H' \times W' \times D}$9 from unoccluded reference viewpoints, reprojects it into the target viewpoint, stitches it with the current rendered image $\hat{F}_s(I)$0, and blends the boundary. The synthetic supervision image is

$\hat{F}_s(I)$1

with fixed $\hat{F}_s(I)$2 along boundary edges, edge bandwidth of 10 pixels, and dilation and erosion applied to patch boundaries before blending. The corresponding fusion loss is

$\hat{F}_s(I)$3

This stage uses only RGB supervision (Chen et al., 24 Sep 2025).

The paper explicitly claims that the method avoids the expensive retraining process characteristic of 2D-prior-guided pipelines, but it does not claim zero optimization. Initial scene reconstruction uses 30,000 iterations with Adam, densifies for the first 15,000 iterations, and prunes anchors with opacity $\hat{F}_s(I)$4 afterward. At inpainting time, GS-RoadPatching performs only 50 additional training iterations with post-fusion, compared with 150 additional training iterations plus 2D inpainting for InFusion (Chen et al., 24 Sep 2025). This suggests a distinction between scene training and editing-time refinement: the method avoids full reconstruction loops, yet still uses a lightweight optimization stage to restore local harmony.

5. Benchmarks, ablations, and interoperability

The driving-scene evaluation uses 23 Waymo Open Dataset sequences, each about 60–90 frames, including 7 multi-dynamic-object scenes, 4 severely occluded scenes, 6 uphill/downhill scenes, 4 complex weather/lighting scenes, and plain-road scenes. General-scene evaluation uses 7 unbounded $\hat{F}_s(I)$5 scenes from 360-USID. On Waymo, the reported metrics are LPIPS, FID, and time; on 360-USID, they are PSNR, SSIM, and LPIPS. Against ProPainter, GScream, InFusion, and StreetGaussians + InFusion, GS-RoadPatching achieves the best FID on Waymo at 74.124, LPIPS of 0.237, and average extra processing time of 64 s. On 360-USID it reports PSNR 17.917, SSIM 0.975, and LPIPS 0.360. The paper also reports a user study with 43 participants in which 88.22% selected the method as best for natural quality and 83.57% selected it as best for clean target removal (Chen et al., 24 Sep 2025).

The ablations are structurally informative. For voxel size $\hat{F}_s(I)$6, 250 cm is best, with LPIPS 0.237 and FID 74.124; 50 cm loses global consistency, while 500 cm introduces noise and artifacts. For descriptors, color-only matching yields LPIPS 0.354 and FID 92.453, DINO yields 0.258 and 81.924, and SuperPoint yields 0.237 and 74.124. For the final editing stage, fusion only gives LPIPS 0.322 and FID 98.537, substitution only gives 0.347 and 93.386, fusion without edge blending gives 0.263 and 82.645, and full substitution plus fusion gives 0.237 and 74.124 (Chen et al., 24 Sep 2025).

The paper uses the term interoperability to denote that the output remains an editable 3D Gaussian scene asset rather than only a set of modified frames. This is operationally significant. The completed scene supports novel-view synthesis, additional editing, and reuse in downstream 3D pipelines. Teaser and edit figures show object removal as well as direct insertion or movement of foreground vehicles. A plausible implication is that GS-RoadPatching is as much an editing framework as an inpainting framework, provided that a sufficiently reconstructed Gaussian scene already exists (Chen et al., 24 Sep 2025).

6. Position within road-scene Gaussian splatting and principal limitations

GS-RoadPatching sits within a broader line of road-scene Gaussian methods but occupies a distinct methodological position. RoGS represents the road as a meshgrid of 2D Gaussian surfels and optimizes geometry, appearance, and semantics over a trajectory-aligned square grid for large-scale road-surface reconstruction (Feng et al., 2024). BEV-GS likewise specializes Gaussian splatting to a road BEV grid and predicts road elevation and texture in a feed-forward manner from a single monocular image (Wu et al., 16 Apr 2025). MagicRoad defines road patching as recovering the clean underlying road surface from occluded urban driving video by combining segmentation-guided video inpainting, semantic-aware HSV harmonization, and planar 2D Gaussian surfels (Peng et al., 31 Jul 2025). By contrast, GS-RoadPatching assumes a reconstructed 3DGS scene and performs completion through 3D retrieval and placement rather than through road-surface-specific reconstruction or 2D inpainted supervision (Chen et al., 24 Sep 2025).

This distinction helps clarify scope. GS-RoadPatching is not primarily a road-surface reconstruction method, a lane-level BEV mapping method, or a dynamic-scene Gaussian renderer. Methods such as GGS focus on generalizable lane-switch novel-view synthesis under large lateral viewpoint changes (Han et al., 2024), and EGSRAL focuses on dynamic/static modeling, grouping, and auto-labeling in large-scale driving-scene Gaussian rendering (Huo et al., 2024). GS-RoadPatching instead addresses the local completion problem after reconstruction: locating a hole or removable object region, retrieving a compatible patch from elsewhere in the scene, and restoring consistency with limited post-fusion optimization (Chen et al., 24 Sep 2025).

The paper states several assumptions and limitations explicitly. The method is not real-time; it is intended for offline high-quality completion and editing. It assumes scene repetitiveness and structural redundancy, road-like manifold structure for BEV search, sufficient reconstruction quality, and at least some perspective supervision in which the background is visible in other frames. It may fail when a large object permanently occludes a unique structure, when reliable references do not exist elsewhere in the scene, or when the scene lacks the repetitive regularities typical of road environments. This suggests that substitutional patching is strongest where road scenes are regular and weakens as scene uniqueness increases, whereas generative methods may retain an advantage when no reusable source patch exists (Chen et al., 24 Sep 2025).