SalientGS: Unified SfM to 3D Gaussian Splatting

• SalientGS is a unified SfM-to-3D Gaussian Splatting pipeline that reconstructs 3D scenes and refines camera poses from unordered image collections.
• It employs an importance-guided MCMC Gaussian allocation mechanism that uses multi-view residuals to direct Gaussian birth and relocation toward underfit regions.
• By integrating pose estimation and rendering, SalientGS achieves state-of-the-art perceptual quality with rapid reconstruction times under a fixed Gaussian budget.

SalientGS is a unified SfM-to-3D Gaussian Splatting pipeline for reconstructing 3D scenes and camera poses from unordered image collections. Its central contribution is importance-guided Markov Chain Monte Carlo Gaussian allocation, which aggregates multi-view residuals into per-Gaussian underfit and redundancy signals and then uses those signals to bias Gaussian birth and relocation toward underfit regions without altering the underlying stochastic gradient Langevin dynamics. In the reported experiments, the method performs end-to-end reconstruction with state-of-the-art perceptual quality and is presented as achieving reconstruction in 15 minutes (Xiong et al., 13 Jul 2026).

1. Concept and problem setting

SalientGS addresses a specific bottleneck in conventional 3DGS pipelines: they typically assume that camera poses and a sparse point cloud are already available, usually from COLMAP, and then optimize a Gaussian scene representation as a separate stage. The method identifies two drawbacks in that arrangement. The first is expensive preprocessing, because exhaustive matching and second-order bundle adjustment can take as long as, or longer than, 3DGS training. The second is a frozen interface between pose estimation and radiance optimization, meaning that pose errors cannot be corrected photometrically and cannot exploit the radiance field representation (Xiong et al., 13 Jul 2026).

The defining idea of SalientGS is therefore not merely Gaussian densification. It is a unification of an SfM front-end, joint pose refinement, and a population-management scheme for Gaussians under a fixed budget. In this formulation, the Gaussian population is managed through an MCMC view inherited from 3DGS-MCMC rather than through standard adaptive density control based on clone/split and opacity pruning. The saliency signal in SalientGS is not a semantic label and not an explicit per-Gaussian “saliency attribute.” Instead, it is operationalized as importance derived from persistent multi-view residuals and contrasted with redundancy derived from consistently low-error coverage.

This design places SalientGS in the class of joint pose-and-rendering methods rather than pose-fixed splatting pipelines. It also distinguishes the method from approaches that depend on a separate learned global pose predictor or on a fully frozen sparse reconstruction. The method still depends on successful SfM initialization, but it uses that initialization as a starting point rather than as an immutable input.

2. Unified SfM-to-3DGS pipeline

The pipeline proceeds from unordered images to a trained 3DGS model in four stages. First, it performs global correspondence estimation by extracting SIFT features, building Fisher Vector descriptors, retrieving top-$k$ nearest neighbors with FAISS, constructing a sparse but connected view graph using $k$NN edges plus a maximum spanning tree, and finally applying RANSAC geometric verification on candidate pairs to obtain the final reliable image-pair set $\mathcal{P}$ (Xiong et al., 13 Jul 2026).

Second, it runs first-order SfM. Intrinsics are estimated with a one-parameter division model, with focal length inferred via essential-matrix singular-value ratio. Global rotations are solved by geodesic minimization on $SO(3)$, translations by direction-only consistency, and the pose estimate is refined using structureless epipolar adjustment. For each matched pair $n$, the essential matrix is vectorized as $\mathbf{e}_n = \mathrm{vec}(E_n)$, and the epipolar loss is

$$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$

where $W_n \in \mathbb{R}^{9 \times 9}$ is a precomputed quadratic form and $Z$ is a normalization factor. The method then triangulates 3D points $\{\mathbf{X}_k\}$ and tracks, while still retaining a structureless formulation for pose refinement.

Third, it initializes 3D Gaussians from the coarse 3D structure and poses. The details are stated to follow standard 3DGS or MCMC practice, with initial Gaussians typically seeded near triangulated points or along frustum rays.

Fourth, it runs joint 3DGS and pose optimization. In the reported training schedule, this stage lasts 30K iterations. During training, the method renders views, computes photometric and SSIM losses, computes a reprojection bundle-adjustment loss on SfM tracks, and updates both Gaussian parameters and pose adjustments. After a 3K-iteration warmup and then every $k$0 steps, it samples $k$1 views, builds multi-view residual maps, aggregates them into per-Gaussian importance and redundancy signals, constructs an importance-weighted sampling distribution, and uses that distribution to guide birth and relocation.

A central consequence of this design is that SfM and splatting are not separate optimization problems. SfM provides an initialization, but the camera poses continue to be refined during splatting, and SfM tracks remain active as a geometric anchor through the bundle-adjustment term.

3. Importance-guided MCMC Gaussian allocation

The importance-guided allocation mechanism is the method’s most distinctive component. At designated training steps, SalientGS computes residual maps across $k$2 sampled views. For each view $k$3, with rendered RGB image $k$4 and ground-truth image $k$5, the per-pixel L1 error is defined as

$k$6

with $k$7 for RGB (Xiong et al., 13 Jul 2026).

Rather than normalizing with min-max scaling, the method uses quantile-based normalization:

$k$8

where $k$9 and $\mathcal{P}$0 are low and high error quantiles, given in the experiments as $\mathcal{P}$1 and $\mathcal{P}$2. It then defines high- and low-error thresholds using the quantiles of $\mathcal{P}$3, with $\mathcal{P}$4.

For a Gaussian $\mathcal{P}$5, let $\mathcal{P}$6 denote its 2D footprint in view $\mathcal{P}$7. The high-error contribution at a pixel $\mathcal{P}$8 is

$\mathcal{P}$9

and the resulting per-Gaussian importance score is

$SO(3)$0

The division by $SO(3)$1 prevents a bias toward large Gaussians, and the factor $SO(3)$2 places the score on a percentage-like $SO(3)$3–$SO(3)$4 scale, with $SO(3)$5 in the default configuration.

Redundancy is defined symmetrically from low-error coverage. The low-error contribution is

$SO(3)$6

and the redundancy score is

$SO(3)$7

This redundancy score is then min-max normalized to $SO(3)$8.

The importance-weighted sampling distribution is derived from the importance score through

$SO(3)$9

followed by normalization. To preserve exploration for underfit Gaussians that may still have low current opacity, the method mixes opacity and importance:

$n$0

with $n$1. This distribution is then used in two population-management operations. In relocation, Gaussians with low opacity or high redundancy are removed from their current location and reassigned near target parents sampled from $n$2. In birth, if the current population is below the budget cap $n$3M, new Gaussians are spawned from parents sampled according to the same distribution.

A frequent misunderstanding is to treat this mechanism as a modification of the underlying SGLD optimizer. The paper explicitly states the opposite: SalientGS does not change the SGLD dynamics. It changes only where Gaussians are added or relocated.

4. Gaussian representation, rendering, and optimization

SalientGS uses a standard 3D Gaussian Splatting representation. Each Gaussian $n$4 has a 3D mean $n$5, a covariance represented through scale and rotation, color coefficients or direct RGB, and an opacity parameter $n$6 mapped through a sigmoid to $n$7. Rendering follows the standard projection, rasterization, depth-ordering, and alpha-compositing pipeline rather than introducing a new renderer (Xiong et al., 13 Jul 2026).

The compositing equation is written as

$n$8

where $n$9 is the Gaussian’s color contribution at the pixel and $\mathbf{e}_n = \mathrm{vec}(E_n)$0 is its opacity contribution. Visibility and opacity remain implicit in the splatting and compositing process, which is why low-opacity Gaussians become natural candidates for relocation.

The training objective combines photometric supervision, bundle-adjustment supervision, and regularization:

$\mathbf{e}_n = \mathrm{vec}(E_n)$1

The photometric term uses an L1-plus-SSIM objective,

$\mathbf{e}_n = \mathrm{vec}(E_n)$2

with $\mathbf{e}_n = \mathrm{vec}(E_n)$3. The geometric anchoring term is a reprojection bundle-adjustment loss on SfM tracks:

$\mathbf{e}_n = \mathrm{vec}(E_n)$4

where $\mathbf{e}_n = \mathrm{vec}(E_n)$5 is the set of 2D observations, $\mathbf{e}_n = \mathrm{vec}(E_n)$6 is the triangulated 3D point for track $\mathbf{e}_n = \mathrm{vec}(E_n)$7, $\mathbf{e}_n = \mathrm{vec}(E_n)$8 is the camera pose, and $\mathbf{e}_n = \mathrm{vec}(E_n)$9 is the projection function. Only camera poses and Gaussians are optimized; the triangulated track points are kept fixed. The reported value is $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$0.

The MCMC interpretation is coupled to SGLD updates of the form

$$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$1

where $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$2. In SalientGS, this stochastic optimization remains unchanged. The importance-guided mechanism is explicitly described as orthogonal to the SGLD step and intended to improve sample efficiency under a fixed Gaussian budget.

5. Empirical performance and computational profile

The reported evaluation covers Mip-NeRF 360 with 9 scenes, Deep Blending with 2 scenes, and Tanks & Temples with 2 scenes, using PSNR, SSIM, and LPIPS as metrics. Competing methods include 3DGS, 3DGS-MCMC, Mini-Splatting, Speedy-splat, Taming-3DGS, DashGaussian, FastGS-big, GloSplat-A, and VGGT-X. SalientGS is reported as the fastest method in the comparison table when prior methods’ COLMAP times are included (Xiong et al., 13 Jul 2026).

Benchmark	SalientGS result	Runtime / budget
Mip-NeRF 360	28.82 dB / 0.853 / 0.148	11.79 min, 1.5M Gaussians
Deep Blending	29.49 / 0.906 / 0.183	10.04 min, 1.5M Gaussians
Tanks & Temples	24.65 / 0.869 / 0.109	10.03 min, 1.5M Gaussians

Across the three benchmarks, the paper reports a cross-benchmark macro-average of approximately $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$3 dB PSNR, $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$4 SSIM, and $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$5 LPIPS. On Mip-NeRF 360, SalientGS is reported as close to GloSplat on quality while being approximately $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$6 faster and using half the Gaussians, and it is described as far more robust on Deep Blending because GloSplat-A and VGGT-X fail catastrophically on the drjohnson scene.

The ablation studies isolate the role of the allocation mechanism. Removing guided birth reduces PSNR to $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$7 dB, removing guided relocation gives $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$8 dB, and removing both to recover vanilla MCMC also yields $$\mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n,$$9 dB with LPIPS $W_n \in \mathbb{R}^{9 \times 9}$0 instead of $W_n \in \mathbb{R}^{9 \times 9}$1. Replacing MCMC with standard adaptive density control reduces performance to $W_n \in \mathbb{R}^{9 \times 9}$2 dB PSNR, $W_n \in \mathbb{R}^{9 \times 9}$3 SSIM, and $W_n \in \mathbb{R}^{9 \times 9}$4 LPIPS. The most severe ablation is the removal of footprint normalization, which causes a reported drop to $W_n \in \mathbb{R}^{9 \times 9}$5 dB and LPIPS $W_n \in \mathbb{R}^{9 \times 9}$6. This establishes the footprint-normalized aggregation over $W_n \in \mathbb{R}^{9 \times 9}$7 as a structurally important part of the method rather than an implementation detail.

A second ablation series addresses pose refinement. If the bundle-adjustment term is removed, performance drops to $W_n \in \mathbb{R}^{9 \times 9}$8 dB and LPIPS worsens to $W_n \in \mathbb{R}^{9 \times 9}$9. If poses are frozen after SfM initialization, performance drops further to $Z$0 dB with $Z$1 SSIM instead of $Z$2. The paper interprets this as showing that photometric refinement provides the main pose gain, while the BA term adds stability.

The initialization quality of the sparse view graph also matters. With retrieval parameter $Z$3, the reported PSNR is $Z$4 dB; with the default $Z$5, $Z$6 dB; with $Z$7, $Z$8 dB; and with $Z$9, $\{\mathbf{X}_k\}$0 dB, including one catastrophic failure. The method therefore reduces but does not eliminate dependence on a viable SfM graph.

Budget-efficiency experiments further support the allocation strategy. Across budget caps from $\{\mathbf{X}_k\}$1K to $\{\mathbf{X}_k\}$2M, importance guidance yields reported PSNR gains of $\{\mathbf{X}_k\}$3–$\{\mathbf{X}_k\}$4 dB. Notably, guided $\{\mathbf{X}_k\}$5M Gaussians at $\{\mathbf{X}_k\}$6 dB outperform vanilla $\{\mathbf{X}_k\}$7M at $\{\mathbf{X}_k\}$8 dB.

6. Position within salient Gaussian splatting

The term “salient” in SalientGS refers to importance-guided allocation under a fixed Gaussian budget rather than to a special anisotropic Gaussian parameterization. That distinction is important because another contemporaneous method, LR-SGS, uses “salient Gaussian splatting” to denote structure-aware Gaussians aligned with edges and planes in self-driving scenes, with LiDAR geometry, LiDAR reflectance, a per-Gaussian reflectance channel, and cross-modal boundary alignment. In LR-SGS, saliency is derived from LiDAR geometric features, LiDAR reflectance edges, and emergent Gaussian anisotropy; in SalientGS, saliency is defined by persistent multi-view underfit and redundancy signals (Chen et al., 13 Mar 2026).

The broader 3DGS-SLAM survey provides a wider taxonomy for this distinction. It states that there is no explicit mention of “SalientGS” in the surveyed 3DGS-SLAM literature, but it identifies a family of methods that are “SalientGS-like” in the sense that they allocate more density or optimization effort to regions judged important by residuals, structure, frequency content, visibility, or feature confidence. In that survey’s language, such methods fall under “Vision-Guided Perception” and feature-based or hybrid tracking, rather than under a single canonical “salient” formulation (Wang et al., 4 Feb 2026).

Against that background, SalientGS occupies a specific position. It is neither a LiDAR-guided salient representation nor a SLAM system built around explicit keypoint masks. Its distinguishing features are a fast optimization-based SfM front-end, explicit reprojection BA during splatting-based pose refinement, and an importance-guided MCMC mechanism that redistributes Gaussian capacity using multi-view residual statistics. A useful correction to a common misconception follows directly from this positioning: SalientGS is not simply a “COLMAP-free densification heuristic.” It is a joint pose-and-representation pipeline whose saliency signal acts on Gaussian population management.

7. Limitations, failure modes, and practical use

The paper explicitly notes several limitations. The first is dependence on successful SfM. If the view graph is too sparse, if the retrieval parameter $\{\mathbf{X}_k\}$9 is too small, or if the scene is severely textureless or geometrically degenerate, initialization quality may be poor and reconstruction can fail. On ETH3D SLAM, the paper reports that 2 of 45 sequences fail SfM and 4 additional sequences fail joint training (Xiong et al., 13 Jul 2026).

The second limitation is the speed–optimization trade-off. Joint pose optimization adds overhead relative to frozen-pose 3DGS training, even though the end-to-end wall-clock remains competitive. The third is the use of Fisher Vectors as hand-crafted global descriptors. The paper notes that under extreme appearance changes or in domains where learned global descriptors such as NetVLAD or MASt3R are more robust, pair selection could suffer. It also reports local failures in texture-poor indoor scenes, including cases in which VGGT-X performs better on particular views even though SalientGS leads on scene-mean LPIPS and macro-averages.

The practical configuration reported by the authors is deliberately fixed across all 13 scenes. The front-end uses a global GMM with $k$00 components for Fisher Vector coding, top-$k$01 retrieval with default $k$02, FAISS-based neighbor search, and an MST-augmented graph for connectivity. The splatting stage uses a Gaussian budget cap $k$03M, a 3K-iteration warmup, importance-statistic recomputation every $k$04 iterations, and the fixed loss weights $k$05 and $k$06. The authors state that code, per-scene measurements, evaluation scripts, and CSV logs are available at https://github.com/Six-Bit-TX/SalientGS.

The future directions identified in the paper follow from these constraints. They include reducing joint optimization overhead, integrating learned descriptors into the retrieval and SfM front-end, and extending the importance-guided allocation scheme to other 3DGS objectives such as surface fidelity. A plausible implication is that SalientGS serves as a baseline for a class of joint pipelines in which pose estimation, Gaussian population management, and rendering are optimized together rather than stitched together through a frozen interface.