GSFixer: Enhanced Sparse-View 3D Reconstruction

• GSFixer is a framework that enhances 3D Gaussian Splatting reconstructions by integrating diffusion-based generative priors with advanced conditioning strategies.
• It employs multi-modal inputs—including text, semantic features, and geometric cues—to restore geometric stability and correct photometric artifacts from sparse views.
• Its modular, iterative pipeline refines renderings for cross-view consistency and artifact reduction, advancing state-of-the-art sparse-view 3D reconstruction.

GSFixer refers to a suite of frameworks and algorithms designed to enhance 3D Gaussian Splatting (3DGS) reconstructions, particularly in the under-constrained and artifact-prone setting of sparse-view scene capture. By integrating diffusion-based generative priors and specialized conditioning strategies, GSFixer aims to restore geometric and photometric fidelity in novel views generated from severely incomplete input observations. Recent variants leverage advances in video diffusion, cross-modal semantic fusion, and training-free distillation to push the state of the art in artifact removal, temporal coherence, and 3D consistency (Yin et al., 13 Aug 2025, Wang et al., 23 Sep 2025, Wei et al., 20 Aug 2025, Song et al., 10 May 2026).

1. Motivation and 3DGS Artifacts

3D Gaussian Splatting parameterizes a scene as a set of $N$ anisotropic Gaussians $\{\mu_i, \Sigma_i, c_i, \eta_i\}$, rendered via differentiable volume compositing with alpha blending. This representation yields high-fidelity and interactive-rate rendering under dense multi-view input. However, in the regime of 3–9 input images, the reconstruction task becomes highly under-constrained, resulting in severe ambiguities. Canonical failure modes of sparse-view 3DGS include:

• Geometric instability: “floating” Gaussians, structural holes, and jittery depth in unobserved regions.
• Photometric artifacts: smeared texture, incorrect color, and large hallucinated regions in occlusions.
• Temporal instability: flickering or inconsistent geometry in video-based novel view synthesis.
• Overfitting: faithful input-view reconstructions but poor generalization to unseen viewpoints.

Conventional regularization (smoothness, weak geometric priors) and feedforward learning alleviate but do not eliminate these problems.

2. Core GSFixer Frameworks and Algorithms

GSFixer introduces a generative correction loop that leverages neural generative priors, typically in the form of conditional diffusion models, to “repair” artifact-laden renderings from sparse-view 3DGS (Yin et al., 13 Aug 2025, Wei et al., 20 Aug 2025). Prototypical workflow:

1. Initial Reconstruction: Construct a baseline 3DGS from sparse input views by direct optimization or feedforward architectures such as WorldMirror or DepthSplat.
2. Rendering: Generate artifact-prone renderings (single frames or multi-frame videos) along desired novel view trajectories.
3. Diffusion-Based Fixing: Employ a trained diffusion model to restore clean, semantically and geometrically consistent images by conditioning on both the artifact view and auxiliary information (reference images, semantic features, or geometric cues).
4. Distillation to 3DGS: Project repaired images back into the 3D representation via differentiable photometric or structure-aware losses. This iterative process adjusts the Gaussians to minimize discrepancies between splatted views and generative corrections.
5. Global Refinement: Optionally, alternate or combine with confidence-based weighting, reprojection checks, or adaptive densification to suppress hallucinations and increase cross-view consistency (Song et al., 10 May 2026, Wang et al., 23 Sep 2025).

The pipeline is modular and can be realized with fully supervised, plug-and-play, or training-free components, depending on application scenario and computational constraints.

3. Diffusion Priors, Reference Conditioning, and Training Protocols

GSFixer’s core innovation lies in its use of reference-guided and multi-modal diffusion architectures.

• Video Diffusion Models: The approach in (Yin et al., 13 Aug 2025) employs a DiT-based video diffusion framework (CogVideoX), trained to denoise artifact sequences conditioned on reference frames. Conditioning signals include text captions, 2D semantic features (DINOv2), and 3D geometric tokens (VGGT). These are fused and injected via cross-attention into the backbone at each diffusion step, ensuring semantic alignment and 3D geometric consistency.
• Dual-Modal Inputs: The framework in (Wei et al., 20 Aug 2025) uses a Stable Diffusion v2 latent backbone, concatenating paired mesh and 3DGS renders with view-specific mask augmentation to facilitate plausible inpainting and structure preservation. Random semantic masks simulate occlusion and sparsity, critical for robustness to missing data.
• Confidence-Aware Priors: The confidence-guided variant in (Song et al., 10 May 2026) introduces per-pixel confidence maps via reprojection-based cross-checking between diffusion-enhanced pseudo-targets and available support views, gating both the loss and Gaussian densification.
• Training Regimes: Fully supervised variants finetune large diffusion backbones on paired data synthesized from artifacted 3DGS renders and ground-truth imagery, often augmenting the training set with semantic and geometric cues. Unsupervised/training-free methods (Wang et al., 23 Sep 2025) employ score distillation using pretrained single-step diffusion models in a self-supervised loop.

4. Loss Functions, Optimization, and Consistency Strategies

GSFixer combines generative restoration losses with classical photometric fit and structure-aware losses:

• Diffusion Model Loss: Latent-space denoising objective, $$L_{\text{diff}} = \mathbb{E}_{t, \epsilon}[\|\epsilon_\theta(z_t, t, c) - \epsilon\|^2]$$, where latent variables, diffusion timestep, and conditioning features comprise the context.
• Photometric and SSIM Reprojection: $$L_{\text{recon}} = \lambda_{\text{L1}}\|I^{\text{rendered}} - I^{\text{target}}\|_1 + \lambda_{\text{SSIM}}(1 - SSIM(\cdot, \cdot))$$, with task-specific weighting.
• Generative Distillation Supervision: $$L_{\text{total}} = L_{\text{recon}} + \lambda(t)L_{\text{gen}}$$, where $L_{\text{gen}}$ aligns rendered novel views to their diffusion-corrected counterparts, and $\lambda(t)$ is typically annealed across optimization.
• Confidence-Weighted Losses: Confidence maps $W_v(p)$ modulate each pixel’s contribution to the total loss and to the Gaussians’ positional/pruning gradients.
• Cross-View Consistency: Iterative global optimization, multi-reference enhancement, and adaptive progressive enhancement (APE) pools information from nearest-neighbor views to improve incoherent or unreliably inpainted regions (Wang et al., 23 Sep 2025).

5. Benchmarks, Quantitative Evaluation, and Qualitative Results

GSFixer has been rigorously evaluated on synthetic and real datasets specifically curated for sparse-view 3DGS artifact restoration, including DL3DV-Res (Yin et al., 13 Aug 2025), DL3DV-10K (Wang et al., 23 Sep 2025), Replica, ScanNet++ (Wei et al., 20 Aug 2025), and large-scale driving scenarios (Waymo, nuScenes, KITTI) (Song et al., 10 May 2026).

Tabulated Comparison (selected metrics, DL3DV-Res, PSNR/SSIM/LPIPS)

Method	PSNR ↑	SSIM ↑	LPIPS ↓
3DGS (Baseline)	13.72	0.410	0.521
GenFusion	14.56	0.453	0.486
Difix3D+	14.14	0.419	0.455
GSFixer	16.72	0.520	0.399

GSFixer achieves consistent improvements across all fidelity and perceptual metrics, additionally yielding higher subject/background consistency on video benchmarks and sharper, more stable sequences.

• ConFixGS achieves up to +3.68dB PSNR gain on Waymo, with FID reduced nearly by half relative to baseline (Song et al., 10 May 2026).
• GSFixer within GSFix3D provides up to +5dB PSNR over single-step DIFIX baselines, with essential gains in inpainting regions and floaters removal thanks to mask augmentation and dual-modal conditioning (Wei et al., 20 Aug 2025).
• FixingGS establishes new state-of-the-art on DL3DV-10K and Mip-NeRF360, producing more coherent reconstructions in held-out and extreme novel views (Wang et al., 23 Sep 2025).

6. Limitations, Open Directions, and Extensions

GSFixer models demonstrate maximal impact when the following conditions are met:

• Input views exhibit moderate artifacts yet cover major geometric elements of the scene (typical for sparse but not degenerate captures).
• Fusion and conditioning signals (semantic, geometric) can be reliably extracted from available reference views.
• The computational burden of diffusion denoising (∼50 steps for video, ∼10 steps for images) is acceptable within the intended application context.

Limitations include:

• Diminished efficacy when geometric coverage is incomplete (entire scene regions unobserved) or when initial 3DGS is highly degraded.
• Additional latency induced by multi-step diffusion sampling.
• Difficulty handling dynamic content, specular/transparent materials, or severe texture homogeneity (Yin et al., 13 Aug 2025, Wang et al., 23 Sep 2025).
• Scene-specific fine-tuning required for optimal performance in certain variants (Wei et al., 20 Aug 2025).

Future research directions include:

• Accelerating reference-guided diffusion via single-step or distillation techniques.
• Integrating temporal priors or multi-scale video-based diffusion for enhanced temporal or patchwise coherence.
• End-to-end optimization coupling 3D field learning with generative priors.
• Addressing generalization to unbounded/360° scenes and highly dynamic environments (Yin et al., 13 Aug 2025, Song et al., 10 May 2026).

7. Related Approaches and Comparative Perspective

GSFixer constitutes a distinct approach relative to prior methods:

• Iterative optimization-based 3DGS: Traditional approaches require 30–50k iterations from scratch and are brittle under sparsity, prone to non-convergent artifacts.
• Single-step diffusion fixers (Difix, DIFIX-3D): Lacking explicit reference integration or cross-view constraints, often lead to hallucinations, local mode collapse, or limited structural correction.
• Plug-and-play confidence-guided repair (ConFixGS): Achieves both qualitative and quantitative advances with minimal backbone modification, rapid optimization (∼14 min per scene), and superior suppression of hallucinated content, distinguishing itself from high-complexity per-scene retraining (Song et al., 10 May 2026).
• Score distillation (FixingGS): Enables fully training-free operation while enforcing geometric consistency, though sensitive to priors’ domain match and capable of moderate hallucination in extreme viewpoints (Wang et al., 23 Sep 2025).

The collective body of GSFixer research underscores the advantage of fusing reference-guided generative priors with explicit cross-view constraints and semantic/geometric fusion, forging a new direction in artifact restoration for sparse-view 3D reconstruction.