Improving Sparse-View 3DGS Generalization via Flat Minima Optimization

Abstract: Recent advances in neural rendering have established 3D Gaussian Splatting (3DGS) as a highly efficient representation for novel view synthesis, enabling fast training and real-time rendering with strong fidelity. However, when supervision is limited to sparse input views, 3DGS tends to overfit to the observed images and generalize poorly to unseen viewpoints. We address this challenge from the perspective of flat minima (FM) optimization, which seeks solutions that remain stable under small parameter perturbations. Viewing Gaussian parameters as trainable weights, we adapt FM principles to the geometric and dynamic nature of 3DGS with a lightweight training framework. Our method regularizes optimization with controlled Gaussian perturbations that account for each Gaussian's anisotropy and the training progress, preserving fine details while improving robustness to sparse-view overfitting. To further stabilize this flat minima optimization process, we introduce periodic reinitialization, which temporarily returns non-positional parameters to their initial states for a short window. Together, these techniques integrate seamlessly into existing 3DGS pipelines without architectural changes. Experiments on LLFF and Mip-NeRF360 datasets demonstrate improved quantitative metrics and perceptual quality under sparse-view supervision, producing reconstructions that are sharper, more stable, and better generalized to novel viewpoints.

• The paper introduces a regularization framework that uses flat minima optimization to address overfitting in sparse-view 3DGS.
• It employs scale-adaptive stochastic perturbations and periodic reinitialization to stabilize Gaussian parameters and enhance reconstruction quality.
• Experimental results demonstrate improved PSNR, SSIM, and robustness against perturbations compared to traditional 3DGS baselines.

Flat Minima Optimization for Sparse-View 3D Gaussian Splatting

Introduction

Sparse-view 3D scene reconstruction with 3D Gaussian Splatting (3DGS) remains fundamentally ill-posed due to limited multi-view supervision. Vanilla 3DGS is susceptible to overfitting, yielding reconstructions that generalize poorly to novel viewpoints. This work directly addresses this limitation by reframing generalization as a problem of seeking flat, robust optima in the 3DGS loss landscape. The proposed framework injects carefully controlled, scale-adaptive perturbations into 3D Gaussian parameters and incorporates periodic parameter reinitialization, both aimed at encouraging convergence to flat minima that remain stable under parameter shifts, thus improving generalization and robustness in highly under-constrained settings.

Methodology

The methodology builds a regularization framework around the principle of flat minima (FM) optimization, as defined in modern deep learning literature. The strategy is structured around two principal mechanisms: stochastic, scale-adaptive perturbation (SAP) and periodic reinitialization of non-positional Gaussian parameters.

The method formally treats the parameters of each Gaussian primitive (position, scale, rotation, color, opacity) as learnable weights. At each training step, the pose of each Gaussian is perturbed by adding Gaussian noise with covariance matched to the primitive’s anisotropic scale and spatial orientation. This SAP mechanism ensures that the magnitude and directionality of noise reflects each Gaussian’s geometric characteristics, preventing underfitting of fine detail and overregularization of small or elongated structures. Perturbation is applied stochastically to individual Gaussians, which efficiently mixes perturbed and unperturbed signals during optimization without the compute cost of explicit dual forward passes, in contrast to prior adversarial or ensemble-based FM approaches.

Figure 1: Flat minima optimization framework for sparse-view 3DGS with SAP and periodic reinitialization designed to regularize optimization toward robust, generalizable solutions.

The perturbation magnitude is linearly annealed from zero to a target noise level over the course of training, preventing destabilization in early optimization. In parallel, scale, rotation, and higher-order SH coefficients are periodically reset to their Structure-from-Motion (SfM) initialization values for a short interval, while opacity resets follow standard 3DGS procedures. This periodic reinitialization acts as an additional form of regularization, transiently freezing certain model degrees of freedom and mitigating overfitting.

Figure 2: Per-iteration procedure: at each optimization step, position parameters are temporarily perturbed according to SAP, the loss is evaluated and gradients are computed, but parameter updates are applied only to the original, pre-perturbed model.

Experimental Analysis

The evaluation is conducted on LLFF and Mip-NeRF360 datasets with 3, 6, 9, 12, and 24 view configurations. Quantitative results consistently show that the proposed flat minima framework either matches or outperforms state-of-the-art baselines (3DGS, DropGaussian, CoR-GS, FSGS, and DNGaussian) in terms of PSNR, SSIM, and LPIPS.

Qualitative results also confirm that competing baselines exhibit characteristic artifacts under sparse supervision: blurred details, spatial misalignment, and structural inconsistency, particularly in areas not visible in the input images. In contrast, the presented SAP+reinit scheme yields sharper, less noisy, and structurally consistent reconstructions on both background and occlusion boundaries, with improved stability under novel view synthesis.

A perturbation robustness analysis further demonstrates that models trained with the proposed framework exhibit sublinear degradation in test PSNR when subjected to parameter perturbations post-training, as opposed to the sharper quality collapse observed in 3DGS and DropGaussian.

Figure 3: Perturbation robustness—smaller test PSNR drops with increasing perturbation magnitude compared to baselines, indicating convergence to flatter minima.

Ablation Studies

The ablation section examines several dimensions of the regularization framework:

• Noise type: Anisotropic, scale-matched perturbations significantly outperform isotropic or fixed-magnitude perturbations.
• Parameter selection: Position perturbations yield the greatest generalization effect, with negligible or negative gains from perturbing scale, rotation, or opacity.
• Perturbation schedule: Linear ramping of noise magnitude outperforms static perturbation, validating the need for scheduled regularization.
• Reinitialization: Periodic return-to-initialization of non-positional parameters consistently reduces overfitting and improves generalization, as evidenced by test PSNR trajectories and ultimate test set accuracy.

  Figure 4: PSNR trajectories showing that periodic reinitialization suppresses overfitting and enables continued test PSNR improvements, confirming its effectiveness as a regularizer.

Plug-in Compatibility and Downstream Integration

Since this approach does not require architectural changes, it is demonstrated to be compatible as a plug-in with various optimization-based, diffusion-augmented, and feed-forward pipelines (e.g., Difix3D+, AnySplat). In all settings, the SAP+reinit framework enables additional accuracy and perceptual quality gains, corroborating its generality and versatility as a sparse-view regularizer for 3DGS.

Implications and Future Directions

The results indicate that optimization-driven regularization via per-primitive, scale-adaptive perturbations and periodic parameter reinitialization offers a robust and efficient solution to the sparse-view generalization problem in 3DGS. The implications are broadly relevant for any optimization-based geometric learning under limited data, as the approach implicitly biases solutions toward those robust against both geometric noise and model parameter disturbance.

On the theoretical side, these findings reinforce the critical role of flat minima for generalization in non-convex, geometric deep models with task-specific, physics-based parameterizations. Practically, the results suggest that similar perturbative regularization schemes can be readily adopted in evolving 3D scene representation frameworks, including hybrid neural/splat-based pipelines and non-photorealistic rendering tasks.

Potential future work includes developing adaptive, data-driven scheduling and perturbation strategies, as well as exploring the effect of flat minima optimization on non-positional Gaussian parameters and investigating synergy with external geometric or learned priors under varying levels of supervision.

Conclusion

This work successfully translates flat minima theory from standard deep neural networks to the 3DGS context, yielding a lightweight, plug-and-play regularization scheme. Scale-adaptive, stochastic perturbation of Gaussian positions combined with periodic parameter reinitialization significantly improves generalization and perceptual quality under extreme sparsity regimes, with negligible computational overhead and without the need for additional priors or network changes. These findings advance the frontier of robust 3D scene reconstruction from sparse views by expanding the toolkit of optimization-based regularization for complex geometric parameterizations.