--- title: 'SalientGS: Unified SfM to 3D Gaussian Splatting' url: https://www.emergentmind.com/topics/salientgs type: topic --- # SalientGS: Unified SfM to 3D Gaussian Splatting SalientGS is a unified SfM-to-3D Gaussian Splatting pipeline for reconstructing 3D scenes and camera poses from unordered image collections. Its central contribution is importance-guided Markov Chain Monte Carlo Gaussian allocation, which aggregates multi-view residuals into per-Gaussian underfit and redundancy signals and then uses those signals to bias Gaussian birth and relocation toward underfit regions without altering the underlying stochastic gradient Langevin dynamics. In the reported experiments, the method performs end-to-end reconstruction with state-of-the-art perceptual quality and is presented as achieving reconstruction in 15 minutes [2607.11285]. ## 1. Concept and problem setting SalientGS addresses a specific bottleneck in conventional 3DGS pipelines: they typically assume that camera poses and a sparse point cloud are already available, usually from COLMAP, and then optimize a Gaussian scene representation as a separate stage. The method identifies two drawbacks in that arrangement. The first is expensive preprocessing, because exhaustive matching and second-order bundle adjustment can take as long as, or longer than, 3DGS training. The second is a frozen interface between pose estimation and radiance optimization, meaning that pose errors cannot be corrected photometrically and cannot exploit the radiance field representation [2607.11285]. The defining idea of SalientGS is therefore not merely Gaussian densification. It is a unification of an SfM front-end, joint pose refinement, and a population-management scheme for Gaussians under a fixed budget. In this formulation, the Gaussian population is managed through an MCMC view inherited from 3DGS-MCMC rather than through standard adaptive density control based on clone/split and opacity pruning. The saliency signal in SalientGS is not a semantic label and not an explicit per-Gaussian “saliency attribute.” Instead, it is operationalized as importance derived from persistent multi-view residuals and contrasted with redundancy derived from consistently low-error coverage. This design places SalientGS in the class of joint pose-and-rendering methods rather than pose-fixed splatting pipelines. It also distinguishes the method from approaches that depend on a separate learned global pose predictor or on a fully frozen sparse reconstruction. The method still depends on successful SfM initialization, but it uses that initialization as a starting point rather than as an immutable input. ## 2. Unified SfM-to-3DGS pipeline The pipeline proceeds from unordered images to a trained 3DGS model in four stages. First, it performs global correspondence estimation by extracting SIFT features, building Fisher Vector descriptors, retrieving top-\(k\) nearest neighbors with FAISS, constructing a sparse but connected view graph using \(k\)NN edges plus a maximum spanning tree, and finally applying RANSAC geometric verification on candidate pairs to obtain the final reliable image-pair set \(\mathcal{P}\) [2607.11285]. Second, it runs first-order SfM. Intrinsics are estimated with a one-parameter division model, with focal length inferred via essential-matrix singular-value ratio. Global rotations are solved by geodesic minimization on \(SO(3)\), translations by direction-only consistency, and the pose estimate is refined using structureless epipolar adjustment. For each matched pair \(n\), the essential matrix is vectorized as \(\mathbf{e}_n = \mathrm{vec}(E_n)\), and the epipolar loss is $$ \mathcal{L}_e = \frac{2}{Z} \sum_{n=1}^{|\mathcal{P}|} \mathbf{e}_n^\top W_n \mathbf{e}_n, $$ where \(W_n \in \mathbb{R}^{9 \times 9}\) is a precomputed quadratic form and \(Z\) is a normalization factor. The method then triangulates 3D points \(\{\mathbf{X}_k\}\) and tracks, while still retaining a structureless formulation for pose refinement. Third, it initializes 3D Gaussians from the coarse 3D structure and poses. The details are stated to follow standard 3DGS or MCMC practice, with initial Gaussians typically seeded near triangulated points or along frustum rays. Fourth, it runs joint 3DGS and pose optimization. In the reported training schedule, this stage lasts 30K iterations. During training, the method renders views, computes photometric and SSIM losses, computes a reprojection bundle-adjustment loss on SfM tracks, and updates both Gaussian parameters and pose adjustments. After a 3K-iteration warmup and then every \(T=500\) steps, it samples \(K=10\) views, builds multi-view residual maps, aggregates them into per-Gaussian importance and redundancy signals, constructs an importance-weighted sampling distribution, and uses that distribution to guide birth and relocation. A central consequence of this design is that SfM and splatting are not separate optimization problems. SfM provides an initialization, but the camera poses continue to be refined during splatting, and SfM tracks remain active as a geometric anchor through the bundle-adjustment term. ## 3. Importance-guided MCMC Gaussian allocation The importance-guided allocation mechanism is the method’s most distinctive component. At designated training steps, SalientGS computes residual maps across \(K\) sampled views. For each view \(j\), with rendered RGB image \(r^j\) and ground-truth image \(g^j\), the per-pixel L1 error is defined as $$ e^j_{u,v} = \frac{1}{D} \sum_{d=1}^{D} \left| r^{j,d}_{u,v} - g^{j,d}_{u,v} \right|, $$ with \(D=3\) for RGB [2607.11285]. Rather than normalizing with min-max scaling, the method uses quantile-based normalization: $$ \mathcal{M}^j = \mathrm{clip}\!\left( \frac{e^j - Q_{\ell}(e^j)}{Q_{h}(e^j) - Q_{\ell}(e^j)}, 0, 1 \right), $$ where \(Q_{\ell}\) and \(Q_h\) are low and high error quantiles, given in the experiments as \(0.05\) and \(0.90\). It then defines high- and low-error thresholds using the quantiles of \(\mathcal{M}^j\), with \((q_{\text{hi}}, q_{\text{lo}}) = (0.9, 0.1)\). For a Gaussian \(\mathcal{G}_i\), let \(\Omega_i^j\) denote its 2D footprint in view \(j\). The high-error contribution at a pixel \(p \in \Omega_i^j\) is $$ \phi^j_{\text{hi}}(p) = \frac{\max\!\big(0, \mathcal{M}^j(p) - \tau^j_{\text{hi}}\big)}{1 - \tau^j_{\text{hi}}}, $$ and the resulting per-Gaussian importance score is $$ s^i_{\text{imp}} = \frac{100}{K} \sum_{j=1}^{K} \frac{1}{|\Omega_i^j|} \sum_{p \in \Omega_i^j} \phi^j_{\text{hi}}(p). $$ The division by \(|\Omega_i^j|\) prevents a bias toward large Gaussians, and the factor \(100\) places the score on a percentage-like \(0\)–\(100\) scale, with \(\tau_{\text{imp}}=5\) in the default configuration. Redundancy is defined symmetrically from low-error coverage. The low-error contribution is $$ \phi^j_{\text{lo}}(p) = \frac{\max\!\big(0, \tau^j_{\text{lo}} - \mathcal{M}^j(p)\big)}{\tau^j_{\text{lo}}}, $$ and the redundancy score is $$ s^i_{\text{red}} = \frac{1}{K} \sum_{j=1}^{K} \frac{1}{|\Omega_i^j|} \sum_{p \in \Omega_i^j} \phi^j_{\text{lo}}(p). $$ This redundancy score is then min-max normalized to \([0,1]\). The importance-weighted sampling distribution is derived from the importance score through $$ \tilde{w}_i = \mathrm{softplus}\!\left( \frac{s^i_{\text{imp}} - \tau_{\text{imp}}}{\tau_{\text{imp}}} \right), $$ followed by normalization. To preserve exploration for underfit Gaussians that may still have low current opacity, the method mixes opacity and importance: $$ w_i \propto \big( (1-\lambda_{\text{mix}})\,\sigma(o_i) + \lambda_{\text{mix}} \big)\,\tilde{w}_i, $$ with \(\lambda_{\text{mix}}=0.05\). This distribution is then used in two population-management operations. In relocation, Gaussians with low opacity or high redundancy are removed from their current location and reassigned near target parents sampled from \(w_i\). In birth, if the current population is below the budget cap \(N_{\text{GS}}=1.5\)M, new Gaussians are spawned from parents sampled according to the same distribution. A frequent misunderstanding is to treat this mechanism as a modification of the underlying SGLD optimizer. The paper explicitly states the opposite: SalientGS does not change the SGLD dynamics. It changes only where Gaussians are added or relocated. ## 4. Gaussian representation, rendering, and optimization SalientGS uses a standard 3D Gaussian Splatting representation. Each Gaussian \(\mathcal{G}_i\) has a 3D mean \(\mathbf{\mu}_i\), a covariance represented through scale and rotation, color coefficients or direct RGB, and an opacity parameter \(o_i\) mapped through a sigmoid to \([0,1]\). Rendering follows the standard projection, rasterization, depth-ordering, and alpha-compositing pipeline rather than introducing a new renderer [2607.11285]. The compositing equation is written as $$ C_{\text{out}} = \sum_i \alpha_i C_i \prod_{k