SalientGS: Unified SfM-to-3DGS with Importance-Guided MCMC Gaussian Allocation

Abstract: Reconstructing 3D scenes from unordered images remains bottlenecked by expensive Structure-from-Motion (SfM) preprocessing and frozen pose interfaces. We present SalientGS, a unified SfM-to-3D Gaussian Splatting (3DGS) pipeline. Its central contribution is importance-guided Markov Chain Monte Carlo (MCMC) Gaussian allocation, which aggregates multi-view residuals into per-Gaussian underfit and redundancy signals. These signals define a smooth importance-weighted sampling distribution that biases both birth and relocation toward underfit regions. This reallocates capacity from well-fit areas without altering the underlying stochastic gradient Langevin dynamics (SGLD). SalientGS achieves end-to-end reconstruction in 15 minutes with state-of-the-art perceptual quality. The supplementary material provides dedicated sections for Per-Scene Qualitative Comparisons and Per-Image Learned Perceptual Image Patch Similarity (LPIPS) Analysis, including failure cases. Code and evaluation scripts are available at https://github.com/Six-Bit-TX/SalientGS.

SalientGS: Unified SfM-to-3DGS with Importance-Guided MCMC Gaussian Allocation

Introduction

SalientGS proposes a joint pipeline for high-quality 3D scene reconstruction from unordered image collections, unifying Structure-from-Motion (SfM) and 3D Gaussian Splatting (3DGS) under a single architecture. The method eliminates bottlenecks associated with traditional two-stage pipelines—most notably, computationally expensive COLMAP-based pose estimation and frozen downstream interfaces—by introducing fast SfM initialization and joint optimization of camera pose and appearance. The principal contribution is an importance-guided Markov Chain Monte Carlo (MCMC) Gaussian allocation scheme, which redirects model capacity toward underfit regions based on multi-view error aggregation while preserving the stochastic Langevin dynamics that underpin MCMC sampling.

SalientGS is rigorously benchmarked across 13 scenes and three datasets, with reproducible schedules and released code. Strong numerical results include achieving the best cross-benchmark macro-average PSNR (27.65 dB), SSIM (0.876), LPIPS (0.147), and fastest end-to-end runtime (10.62 min) within a constrained budget ($1.5\times 10^6$ Gaussians). The pipeline fundamentally alters the standard practice by electronically coupling fast SfM and appearance-based refinement, facilitating efficient and accurate reconstruction without reliance on costly, frozen preprocessing (see Figure 1).

Figure 1: Quality vs. Speed vs. Model Size on Mip-NeRF 360, exhibiting SalientGS's Pareto-optimal tradeoff across perceptual quality and runtime with 1.5M Gaussians and zero COLMAP preprocessing.

Methodology

Global Matching and Sparse Correspondence

SalientGS employs retrieval-based image pair selection using Fisher Vector (FV) descriptors, enforcing global connectivity through Maximum Spanning Tree (MST) augmentation. This procedure yields a sparsely connected, robust matching graph that significantly reduces complexity compared to $O(N^2)$ exhaustive matching, while preserving reliable initialization for downstream SfM.

Figure 2: Matching graph construction: SIFT features encoded to Fisher Vectors, top-k candidate retrieval, MST connectivity, and RANSAC verification ensure efficient yet robust pair selection.

First-Order SfM and Pose Initialization

The pipeline leverages FastMap to achieve first-order, structureless SfM optimization with per-step cost independent of 3D point count. Camera pose estimation proceeds via hierarchical focal length search, SO(3) rotation minimization, and translation direction consistency, all executed using fused CUDA kernels for performance.

Importance-Guided MCMC Gaussian Allocation

SalientGS introduces error-driven importance and redundancy scores for each Gaussian, computed by aggregating robustly normalized multi-view L1 reconstruction error per-pixel. Importance-weighted sampling then guides both Gaussian birth and relocation, reallocating model capacity to persistent underfit regions and recycling redundant primitives.

Figure 3: Multi-view error aggregation yields importance and redundancy scores, biasing birth and relocation toward underfit regions within the MCMC framework.

This allocation operates as a heuristic overlay atop SGLD-based population dynamics, preserving theoretical convergence properties. Empirical evidence demonstrates improved sample efficiency and quality at fixed budgets.

Joint Pose and 3DGS Optimization

Joint optimization proceeds by simultaneously refining camera poses and 3D Gaussian parameters using a composite loss function, comprising photometric error, SSIM, and bundle adjustment (BA) reprojection error on triangulated tracks. This ensures geometric coherence during appearance optimization and mitigates pose drift.

Figure 4: Unified pipeline: candidate pair retrieval, MST matching graph, first-order SfM, and joint pose-appearance 3DGS optimization, concentrating capacity on salient regions without sacrificing geometric anchoring.

Experimental Evaluation

Quantitative Results

Across Mip-NeRF 360, Deep Blending, and Tanks & Temples, SalientGS achieves the highest macro-aggregated metrics in PSNR, SSIM, and LPIPS among all compared pipelines, outperforming both COLMAP-dependent and state-of-the-art pose-optimizing baselines. Strong overall consistency is observed: SalientGS ranks in the top two on all tested datasets and avoids catastrophic failures observed in GloSplat-A and VGGT-X.

Figure 5: Selected-view qualitative comparison, capturing localized detail restoration and highlighting per-view LPIPS differences between SalientGS and baselines.

SalientGS also demonstrates unmatched runtime efficiency, with near-linear scaling and up to 23$\times$ speedup over COLMAP in large-scale SfM initialization.

Figure 6: SfM runtime comparison: SalientGS achieves up to 23$\times$ speedup versus COLMAP, maintaining near-linear scaling with increasing image count.

Ablations

Ablation studies confirm that importance-guided MCMC allocation yields a 0.10 dB PSNR gain and 0.001 LPIPS reduction over vanilla MCMC at 1.5M Gaussians. Birth guidance provides marginally greater improvement than relocation alone. Footprint normalization is critical; its absence degrades PSNR by over 6 dB. Standard clone/split ADC is outperformed by the multi-view allocation approach, supporting distributed capacity optimization.

Joint pose optimization further yields up to 0.5 dB PSNR gain, with photometric refinement providing major recovery and BA loss adding incremental improvement. Sensitivity analyses reveal that smaller retrieval ($k=5$) can induce catastrophic failures, underscoring the necessity of well-connected matching graphs.

Pose Evaluation

Rigorous evaluation on ETH3D SLAM with ground-truth poses demonstrates that joint optimization consistently improves pose accuracy across all metrics, especially when initialized from weaker first-order SfM. For 45 sequences, joint refinement improves [email protected] by $+5.6$ (FastMap), outperforming even GLOMAP despite its inherently stronger initialization. This validates the design: pairing fast initialization with joint refinement achieves superior pose accuracy.

Implications and Future Directions

SalientGS offers a reproducible, efficient alternative to classical two-stage pipelines and demonstrates that heuristic, error-driven population management enhances sample efficiency without theoretical compromise. The tightly coupled architecture enables rapid, high-quality 3DGS reconstructions from arbitrary unordered image sets, with implications for VR/AR, robotics, large-scale mapping, and automated scene digitization.

Practically, SalientGS reduces preprocessing bottlenecks and hardware requirements, contributing to scalable large-scene reconstruction. Theoretical implications include validated gains from importance-driven stochastic population management and joint geometric-photometric optimization.

Future developments may explore learned pair retrieval for robust matching under severe appearance variation, further optimization of joint training schedules, and extension to mesh-based or hybrid volume/splat representations. Ensuring robustness in degenerate camera configurations and minimizing training overhead relative to frozen-pose methods are open avenues.

Conclusion

SalientGS establishes a unification of SfM and 3DGS with importance-guided MCMC allocation, yielding the strongest cross-benchmark quality and runtime efficiency under rigorous reproducibility constraints. The pipeline sets a new technical baseline for joint pose and appearance refinement with error-driven model capacity allocation, while remaining susceptible to upstream initialization failures and pose optimization overhead. The framework provides both practical and theoretical enhancements to automated 3D scene reconstruction, with clear prospects for future algorithmic evolution and application expansion (2607.11285).