Rethinking Pose Refinement in 3D Gaussian Splatting under Pose Prior and Geometric Uncertainty

Abstract: 3D Gaussian Splatting (3DGS) has recently emerged as a powerful scene representation and is increasingly used for visual localization and pose refinement. However, despite its high-quality differentiable rendering, the robustness of 3DGS-based pose refinement remains highly sensitive to both the initial camera pose and the reconstructed geometry. In this work, we take a closer look at these limitations and identify two major sources of uncertainty: (i) pose prior uncertainty, which often arises from regression or retrieval models that output a single deterministic estimate, and (ii) geometric uncertainty, caused by imperfections in the 3DGS reconstruction that propagate errors into PnP solvers. Such uncertainties can distort reprojection geometry and destabilize optimization, even when the rendered appearance still looks plausible. To address these uncertainties, we introduce a relocalization framework that combines Monte Carlo pose sampling with Fisher Information-based PnP optimization. Our method explicitly accounts for both pose and geometric uncertainty and requires no retraining or additional supervision. Across diverse indoor and outdoor benchmarks, our approach consistently improves localization accuracy and significantly increases stability under pose and depth noise.

• The paper introduces UGS-Loc by integrating Monte Carlo sampling of pose priors to address uncertainties in 3D Gaussian Splatting pipelines.
• It employs Fisher Information-based geometric uncertainty to weight depth correspondences, achieving up to 40% improvement in translation error.
• The method demonstrates robust performance on various benchmarks, enabling accurate localization for AR/VR, robotics, and autonomous navigation without retraining.

Uncertainty-Aware Pose Refinement for 3D Gaussian Splatting

Introduction and Motivation

This work proposes a principled solution to the persistent robustness weaknesses of 3D Gaussian Splatting (3DGS)-based visual localization pipelines. The authors identify that state-of-the-art pose refinement pipelines leveraging 3DGS, while benefiting from differentiable rendering, remain acutely sensitive to two major forms of uncertainty: (i) the quality of the initial pose prior, and (ii) spatially-varying reliability of the rendered geometry. Standard pipelines process pose refinement as a deterministic optimization starting from a single pose prior and treat all rendered depths as equally reliable across the scene. These assumptions break down under realistic conditions where pose priors are noisy (frequently the output of retrieval, regression, or SCR systems) and geometric reconstruction is imperfect due to occlusions, view sparsity, or dynamic content.

The proposed Uncertainty-aware Gaussian Splatting Localization (UGS-Loc) directly addresses these deficiencies by explicitly modeling both pose and geometric uncertainties. This approach introduces Monte Carlo pose hypothesis sampling and incorporates Fisher Information-based geometric uncertainty during the PnP-RANSAC optimization, leading to robust relocalization unaffected by the shortcomings of deterministic pipelines.

Figure 1: The UGS-Loc framework integrates pose prior uncertainty via Monte Carlo sampling and geometric uncertainty fields for robust, retraining-free localization.

Methodological Innovations

Monte Carlo Pose Sampling for Prior Uncertainty

UGS-Loc reframes the pose prior as a distribution rather than a point estimate. It draws on Monte Carlo Localization (MCL), sampling multiple pose hypotheses ("particles") from the initial estimate, refining each, and assigning importance weights based on correspondence confidence and geometric reliability. The importance weighting eschews photometric errors in favor of the joint metric comprising match confidence and Fisher Information-derived geometric uncertainty, both of which show strong empirical correlation with pose accuracy.

This strategy provides substantial robustness: if the initial prior is in a region of high uncertainty or poor reconstruction, only a subset of particles—those likely to be closer to the true pose—will accumulate high weight and be propagated forward.

Figure 2: Pipeline comparison—deterministic 3DGS-based refinement (a) is vulnerable to bad priors and geometric anomalies, while UGS-Loc (b) mitigates these issues via diversity in pose hypotheses and uncertainty-weighted sampling.

Fisher Information-Based Geometric Uncertainty

The geometric reliability of 3DGS-rendered depths is inherently heterogeneous due to the explicit representation's limitations and scene variability. The authors extend FisherRF to compute spatially-varying uncertainty fields via Fisher Information, which, for each rendered depth, quantify its reliability with respect to camera pose estimation.

During pose refinement, the UGS-Loc PnP-RANSAC leverages these uncertainty maps to explicitly bias correspondence selection and consensus maximization toward geometrically reliable regions, suppressing outliers emanating from poor scene reconstruction or dynamic content.

Figure 3: Visualization demonstrates the inverse correlation between aggregated matcher confidence (a,c), geometric uncertainty (b,d), and translation error; higher confidence and lower uncertainty yield lower pose error.

Figure 4: Uncertainty-based PnP naturally suppresses correspondences in unreliable regions, improving inlier selection and downstream pose estimation.

Empirical Results

Benchmarks and Numerical Performance

UGS-Loc is evaluated on the 7Scenes, 12Scenes, and Cambridge Landmarks benchmarks. The results demonstrate clear, consistent gains over both initial pose priors (APR, SCR) and prior 3DGS-based refinement pipelines such as GS-CPR. Across indoor and outdoor datasets, the key findings are:

• Substantial reduction in median translation and rotation errors compared to deterministic GS-CPR: Up to 40% improvement over prior poses; further 20% reduction over GS-CPR for some pose estimators.
• Superior recall at stringent error thresholds (e.g., $2\,\text{cm}, 2^\circ$): UGS-Loc achieves 95.1% or higher on 7Scenes compared to 86.1% for the best GS-CPR variant under comparable settings.
• Convergence efficiency: Most of the performance gains are realized in only two iterations with a moderate particle count ($m=8$), making the method computationally tractable.

  Figure 5: Qualitative localization on the 7-Scenes dataset. Rendered views from UGS-Loc (bottom-right in each triplet) show closer alignment to the ground-truth than both pose priors and GS-CPR baseline refinements.

Analysis of Uncertainty Integration

Ablation studies unambiguously attribute the gains to both pipeline components: Monte Carlo refinement (MCR) and Uncertainty-based PnP (UPnP) each provide independent benefits, and their combination achieves the best results. For example, on Cambridge Landmarks, geometric uncertainty-based PnP alone improves median translation error by ≈11%; MCR delivers larger improvements due to enhanced recovery from poor priors. Increasing the number of MC particles improves robustness to initialization, confirming that the approach leverages pose diversity to avoid converging to poor local minima.

Figure 6: UGS-Loc outperforms iterative GS-CPR in both translation and rotation error convergence on the Cambridge Landmarks Church scene, showing the advantage of uncertainty modeling over simple deterministic iterations.

Generalization: Image Retrieval Priors and Matching Modules

UGS-Loc's robustness extends to scenarios where only image retrieval supplies the pose prior—a setting previously vulnerable to catastrophic errors when the top-1 retrieved image is not spatially proximate to the query. By leveraging top-K retrievals as MC particles, UGS-Loc down-weights incorrect hypotheses and converges to the true pose even when ambiguous retrievals would fail deterministically.

Figure 7: With image retrieval priors, UGS-Loc's use of multiple top-K hypotheses enables correct localization in ambiguous scenarios, unlike single-hypothesis pipelines.

Figure 8: UGS-Loc corrects a failure case arising from an incorrect Top-1 retrieval—multiple hypotheses (blue boxes) allow recovery of correct pose.

Furthermore, UGS-Loc is agnostic to the choice of 2D matching module, with competitive performance achieved using both dense matchers (e.g., MASt3r) and lightweight alternatives (e.g., SuperPoint + LightGlue).

Qualitative Evaluation

Rich qualitative results (Figures 9–11) across benchmarks illustrate the visual correspondence between the rendered views from UGS-Loc and ground truth poses, highlighting the method's robustness to both pose prior and geometric uncertainty. Visual inspection confirms reduced misalignment and correction of errors persistent in previous refinement methods, particularly in challenging scenes with significant occlusions or dynamic content.

Implications and Future Directions

UGS-Loc establishes a new standard for robust, retraining-free, and uncertainty-aware pose refinement in pipelines that leverage explicit neural scene representations. Its generality—achieving state-of-the-art performance with various priors and correspondence modules—positions it as a strong baseline for both practical deployment and further research.

Practically, UGS-Loc facilitates accurate AR/VR localization, robotics navigation, and autonomous driving in dynamic environments, accommodating inputs from uncalibrated retrieval systems and handling scenes with significant geometric ambiguities.

Theoretically, this work opens directions for even deeper probabilistic modeling, such as joint uncertainty propagation from geometry, correspondence, and photometric signals; integration with active mapping and autonomous exploration frameworks leveraging explicit reliability measures; and development of hybrid pipelines fusing 3DGS with learned priors for fully end-to-end uncertainty-aware localization.

Conclusion

This paper rigorously diagnoses and addresses the critical limitations of deterministic pose refinement in 3DGS pipelines. By integrating Monte Carlo pose sampling and Fisher Information-based geometric uncertainty into the correspondence-driven PnP optimization loop, UGS-Loc achieves state-of-the-art performance with high robustness to both pose and geometric ambiguities. Comprehensive experiments and ablations demonstrate the theoretical soundness and practical efficacy of probabilistic, uncertainty-aware design for visual relocalization tasks, setting a new benchmark for future research in neural scene representations and robust camera localization.