SplatHLoc: Hierarchical Gaussian Localization

• SplatHLoc is a visual localization paradigm that replaces SfM point clouds with continuous, renderable Gaussian maps for unified feature representation.
• It integrates adaptive retrieval, virtual view synthesis, and hybrid coarse-to-fine matching to achieve precise 2D–3D pose recovery.
• Empirical evaluations demonstrate superior accuracy and efficiency over traditional pipelines, proving effective in both indoor and outdoor environments.

SplatHLoc denotes a splat-based form of hierarchical visual localization and relocalization in which the scene is represented by Gaussian primitives rather than an SfM point cloud. In its broad sense, it refers to an HLoc-like pipeline that preserves global retrieval, local geometric verification, and PnP-based pose recovery, but replaces sparse points and stored per-point descriptors with a renderable Gaussian map that can support localization and view synthesis within a unified representation. This interpretation is already explicit in the description of SplatLoc, which states that its hierarchical 3DGS-based pipeline is “precisely what one would call ‘SplatHLoc’,” and it becomes a named framework in the later Feature Gaussian Splatting formulation of hierarchical visual relocalization with adaptive nearest-view synthesis and hybrid matching (Zhai et al., 2024, Tao et al., 31 Mar 2026).

1. Conceptual origin and relation to HLoc

Hierarchical localization classically proceeds by retrieving candidate database images, establishing local matches, producing 2D–3D correspondences against an SfM model, and estimating pose with PnP+RANSAC. The motivation for SplatHLoc is that point-based hierarchical pipelines are efficient and geometrically well constrained, but they depend on sparse image observations and cannot natively support high-quality novel-view rendering. Conversely, NeRF-style approaches can render but are slow to train and render, and point-based neural maps with stored descriptors can incur large memory footprints (Zhai et al., 2024, Tao et al., 31 Mar 2026).

Within that context, SplatLoc and SplatHLoc occupy two closely related positions. SplatLoc is a 3D Gaussian Splatting-based visual localization method for augmented reality that builds a compact splat map, learns unbiased scene-specific 3D descriptors on demand, selects salient 3D landmarks, and performs 2D–3D matching followed by PnP+RANSAC. The later SplatHLoc framework formalizes hierarchical visual relocalization on top of Feature Gaussian Splatting, adding adaptive nearest-view synthesis and a hybrid coarse-to-fine matcher that exploits a distinction between Gaussian-rendered features and image-extracted fine features (Zhai et al., 2024, Tao et al., 31 Mar 2026).

Aspect	SplatLoc	SplatHLoc
Scene map	3DGS with on-demand 3D descriptor decoder	FGS rendering color, depth, and features
Retrieval	NetVLAD reference retrieval	MixVPR retrieval plus adaptive virtual-view synthesis
Local verification	2D query descriptors to 3D splat descriptors	Coarse rendered-feature matching, then fine semi-dense image matching

A common simplification is to treat SplatHLoc as merely HLoc with a rendered database. That is incomplete. The 2026 formulation replaces sparse point visibility with a continuous renderable feature field and uses synthesized virtual candidates when the original database lacks a nearby viewpoint. The 2024 precursor is likewise not only a rendering system; it performs direct 2D–3D matching to splat-anchored descriptors decoded from 3D positions rather than storing a descriptor vector for every primitive (Zhai et al., 2024, Tao et al., 31 Mar 2026).

2. Scene representations and feature parameterization

The 2026 framework is built on Feature Gaussian Splatting (FGS). Each Gaussian primitive $i$ has a 3D center $x_i \in \mathbb{R}^3$, rotation quaternion $q_i \in \mathbb{R}^4$, anisotropic scale $s_i \in \mathbb{R}^3$, opacity $\alpha_i \in \mathbb{R}$, color $c_i \in \mathbb{R}^3$, and feature $f_i \in \mathbb{R}^d$, with $d=64$ during training and decoding to $C=256$. Projection follows the pinhole model

$\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$

with $x_i \in \mathbb{R}^3$0. The screen-space contribution of splat $x_i \in \mathbb{R}^3$1 at pixel $x_i \in \mathbb{R}^3$2 is

$x_i \in \mathbb{R}^3$3

and normalized compositing yields

$x_i \in \mathbb{R}^3$4

Depth is rendered analogously. The implementation uses the efficient rasterization and compositing strategy of 3DGS via gsplat, including tile-based splat binning and front-to-back alpha compositing (Tao et al., 31 Mar 2026).

FGS is trained jointly for color and dense features. A SuperPoint encoder $x_i \in \mathbb{R}^3$5 produces dense image features $x_i \in \mathbb{R}^3$6. The FGS map renders low-dimensional features $x_i \in \mathbb{R}^3$7 with $x_i \in \mathbb{R}^3$8, and a scene-specific $x_i \in \mathbb{R}^3$9 convolutional decoder $q_i \in \mathbb{R}^4$0 upsamples them to $q_i \in \mathbb{R}^4$1. The loss is

$q_i \in \mathbb{R}^4$2

with

$q_i \in \mathbb{R}^4$3

where $q_i \in \mathbb{R}^4$4 and $q_i \in \mathbb{R}^4$5 (Tao et al., 31 Mar 2026).

The 2024 precursor uses a different feature construction strategy. Instead of storing per-primitive descriptors, SplatLoc first lifts SuperPoint features from multiple posed views into a TSDF-aligned 3D volume $q_i \in \mathbb{R}^4$6, then learns a multi-resolution hash-grid encoding and a small MLP $q_i \in \mathbb{R}^4$7 such that

$q_i \in \mathbb{R}^4$8

defining a scene-specific function $q_i \in \mathbb{R}^4$9. Its “unbiased” designation is tied to the supervision mechanism: the decoder is trained against a lifted 3D feature field on sampled surface points, thereby avoiding alpha-blended supervision that would otherwise bias 3D descriptors toward explaining blended 2D observations rather than precise 2D–3D correspondences (Zhai et al., 2024).

This distinction is significant. FGS renders dense, multi-view consistent feature maps directly for hierarchical relocalization, whereas SplatLoc predicts descriptors at 3D locations for direct 2D–3D matching. Both strategies exploit the compactness and rasterization-friendliness of Gaussian splats, but they operationalize the feature layer differently.

3. Adaptive retrieval and nearest-view synthesis

SplatHLoc’s retrieval stage is designed around the observation that database images may be unevenly distributed, so the nearest retrieved image can still be too far from the query viewpoint to yield sufficient inlier geometry. The online pipeline therefore begins with coarse retrieval, followed by conditional virtual-view synthesis when geometric verification indicates that viewpoint support is inadequate (Tao et al., 31 Mar 2026).

The coarse stage computes a global descriptor $s_i \in \mathbb{R}^3$0 using MixVPR, retrieves the top-$s_i \in \mathbb{R}^3$1 database images, and runs geometric verification with SuperPoint + LightGlue. Candidates are scored by geometric-verification inlier count,

$s_i \in \mathbb{R}^3$2

and the best coarse pose is

$s_i \in \mathbb{R}^3$3

To reduce redundancy, geometric verification is performed only on every 10th coarse-retrieved image. If the best inlier count $s_i \in \mathbb{R}^3$4 is below the threshold $s_i \in \mathbb{R}^3$5, SplatHLoc samples perturbed poses around the best coarse candidate, renders $s_i \in \mathbb{R}^3$6 virtual RGB frames from the FGS map, builds a temporary retrieval database from them, retrieves the top-$s_i \in \mathbb{R}^3$7 virtual candidates, and re-runs geometric verification to pick the nearest view (Tao et al., 31 Mar 2026).

The default hyperparameters are $s_i \in \mathbb{R}^3$8, $s_i \in \mathbb{R}^3$9 indoor and $\alpha_i \in \mathbb{R}$0 outdoor, $\alpha_i \in \mathbb{R}$1, perturbation range $\alpha_i \in \mathbb{R}$2, and translation range $\alpha_i \in \mathbb{R}$3 m indoor and $\alpha_i \in \mathbb{R}$4 m outdoor. The inlier thresholds are $\alpha_i \in \mathbb{R}$5 indoor and $\alpha_i \in \mathbb{R}$6 outdoor. On the Stairs scene, Normal or Random perturbations outperform Uniform, and $\alpha_i \in \mathbb{R}$7–$\alpha_i \in \mathbb{R}$8 balances recall and runtime (Tao et al., 31 Mar 2026).

This stage differentiates SplatHLoc from both classical HLoc and pre-rendered view augmentation. The rendered candidates are query-adaptive rather than fixed. That avoids the storage and search overhead associated with augmenting the database with large numbers of pre-rendered keyframes, while directly addressing large viewpoint gaps.

4. Hybrid coarse-to-fine matching and pose recovery

A central design observation in SplatHLoc is that Gaussian-rendered features and image-extracted features have different strengths. Rendered features are stronger for coarse, patch-level alignment because they encode multi-view information and avoid re-extracting descriptors from rendered RGB, whereas image-extracted features from a semi-dense or dense matcher are stronger for fine, pixel-accurate alignment. The method therefore uses a hybrid matcher rather than a single modality throughout (Tao et al., 31 Mar 2026).

In the coarse stage, the query encoder produces $\alpha_i \in \mathbb{R}$9 at reduced resolution $c_i \in \mathbb{R}^3$0, and the selected real or virtual reference view provides a rendered feature map $c_i \in \mathbb{R}^3$1 on the same grid. Similarity is computed as

$c_i \in \mathbb{R}^3$2

followed by directional softmaxes

$c_i \in \mathbb{R}^3$3

with mutual-nearest-neighbor filtering and threshold $c_i \in \mathbb{R}^3$4. In the fine stage, JamMa extracts semi-dense fine features from the query image $c_i \in \mathbb{R}^3$5 and the rendered RGB image $c_i \in \mathbb{R}^3$6 at resolution $c_i \in \mathbb{R}^3$7. For each coarse match, SplatHLoc crops local $c_i \in \mathbb{R}^3$8 windows, computes local correlations, and applies JamMa’s decoder with MNN and sub-pixel refinement to obtain refined 2D–2D correspondences $c_i \in \mathbb{R}^3$9 (Tao et al., 31 Mar 2026).

Pose recovery lifts the 2D–2D matches to 2D–3D using rendered depth: $f_i \in \mathbb{R}^d$0 An initial pose is then estimated with PoseLib EPnP in RANSAC, followed by robust nonlinear refinement on inliers: $f_i \in \mathbb{R}^d$1 SplatHLoc iteratively re-renders from the current estimate, repeats hybrid matching and PnP, and runs $f_i \in \mathbb{R}^d$2 iterations for indoor datasets and $f_i \in \mathbb{R}^d$3 for outdoor datasets, with early stopping if pose updates are small or inlier counts saturate (Tao et al., 31 Mar 2026).

The precursor SplatLoc uses a different local stage. Query SuperPoint descriptors $f_i \in \mathbb{R}^d$4 are matched directly to decoded 3D Gaussian descriptors $f_i \in \mathbb{R}^d$5, optionally with mutual consistency or a ratio test, and pose is estimated with RANSAC+PnP using the weighted reprojection objective

$f_i \in \mathbb{R}^d$6

That formulation is closer to classical hierarchical 2D–3D localization, whereas the 2026 framework inserts a rendered-reference 2D–2D stage before lifting matches to 3D (Zhai et al., 2024).

5. Training protocol, efficiency, and empirical performance

SplatHLoc trains its FGS map from training images by first building an SfM model to initialize 3D Gaussians and then optimizing the FGS representation for $f_i \in \mathbb{R}^d$7k steps with learning rate $f_i \in \mathbb{R}^d$8, SuperPoint features with $f_i \in \mathbb{R}^d$9, rendered feature channels $d=64$0, and a $d=64$1 convolutional decoder. The implementation follows 3DGS-style splat learning in gsplat, and sky or dynamic regions are masked for Cambridge (Tao et al., 31 Mar 2026).

The efficiency claims are quantitative. On Chess, the reported map size is $d=64$2 MB for SplatHLoc versus $d=64$3 MB for STDLoc; mapping time is approximately $d=64$4 min versus approximately $d=64$5 min; and peak GPU memory is approximately $d=64$6 GB versus approximately $d=64$7 GB. On 7-Scenes, with both methods using four refinement rounds, SplatHLoc’s iterative refinement is nearly $d=64$8 faster than STDLoc because it renders low-dimensional features and performs fine matching at half resolution. Initialization time is reported as comparable, and SplatHLoc avoids training scene-specific detectors or sampling Gaussian spheres (Tao et al., 31 Mar 2026).

The reported rendering fidelity is dataset dependent but operationally sufficient for retrieval and matching. Indoor PSNR is approximately $d=64$9–$C=256$0 dB; outdoor PSNR is approximately $C=256$1–$C=256$2 dB. The paper explicitly notes that this is sufficient for retrieval and matching despite challenging outdoor illumination (Tao et al., 31 Mar 2026).

Across standard benchmarks, SplatHLoc reports the following main results. On 7-Scenes, HLoc/SP+SG achieves $C=256$3 cm/deg, SplatHLoc$C=256$4 achieves $C=256$5, RAPref reports $C=256$6, LoGS/STDLoc report $C=256$7, and SplatHLoc reaches $C=256$8, which is the best overall average. Per-scene bests include $C=256$9 on Chess, $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$0 on Heads, and $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$1 on Stairs. On 12-Scenes, SplatHLoc reports $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$2 with $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$3; ACE+GS-CPR reports $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$4 with $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$5, and DSAC* reports $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$6 with $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$7. On Cambridge Landmarks, SplatHLoc reports average $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$8 cm/deg, improving over LoGS at $\hat{u} = \pi(K, T, X) = \Pi(K [R|t] X),$9 and STDLoc at $x_i \in \mathbb{R}^3$00, with per-scene values of Court $x_i \in \mathbb{R}^3$01, College $x_i \in \mathbb{R}^3$02, Hospital $x_i \in \mathbb{R}^3$03, Shop $x_i \in \mathbb{R}^3$04, and Church $x_i \in \mathbb{R}^3$05 (Tao et al., 31 Mar 2026).

The ablation evidence isolates the two core contributions. On Stairs, a baseline using MixVPR + SuperPoint + LightGlue yields $x_i \in \mathbb{R}^3$06 and $x_i \in \mathbb{R}^3$07. Adding adaptive retrieval changes this to $x_i \in \mathbb{R}^3$08 and $x_i \in \mathbb{R}^3$09 $x_i \in \mathbb{R}^3$10. Hybrid matching alone gives $x_i \in \mathbb{R}^3$11 and $x_i \in \mathbb{R}^3$12. Combining both yields $x_i \in \mathbb{R}^3$13 and $x_i \in \mathbb{R}^3$14, a $x_i \in \mathbb{R}^3$15 improvement over the baseline. In the matching ablation, the FGS-only matcher obtains $x_i \in \mathbb{R}^3$16 and $x_i \in \mathbb{R}^3$17, while replacing only the fine stage with JamMa improves this to $x_i \in \mathbb{R}^3$18 and $x_i \in \mathbb{R}^3$19; replacing the fine stage with ELoFTR gives $x_i \in \mathbb{R}^3$20 and $x_i \in \mathbb{R}^3$21 (Tao et al., 31 Mar 2026).

6. Precursor mechanisms, limitations, and research directions

The term SplatHLoc is also useful for understanding SplatLoc as a precursor architecture. SplatLoc builds a 3DGS map for both rendering and localization, trains an unbiased scene-specific descriptor decoder, selects a small subset of salient landmarks, and localizes by matching 2D image descriptors to 3D descriptors on splats followed by PnP+RANSAC. Its landmark-selection score is

$x_i \in \mathbb{R}^3$22

where $x_i \in \mathbb{R}^3$23 is the learned landmark probability, $x_i \in \mathbb{R}^3$24 is a max viewing-baseline angle across training views, and $x_i \in \mathbb{R}^3$25 is a geometry-consistency term derived from multi-view distances to observed surface points. Selection uses a greedy, saliency-aware farthest-point-like algorithm to produce a spatially dispersed set of high-quality landmarks. For key Gaussians, SplatLoc also freezes positions and applies the regularizer

$x_i \in \mathbb{R}^3$26

which limits anisotropy and center drift. Empirically, this reduces median pose error by about $x_i \in \mathbb{R}^3$27 cm and $x_i \in \mathbb{R}^3$28 on selected Replica scenes (Zhai et al., 2024).

SplatLoc’s efficiency and rendering results explain why it serves as a conceptual antecedent. On 12-Scenes “manolis,” it reports $x_i \in \mathbb{R}^3$29 minutes training, a $x_i \in \mathbb{R}^3$30 MB model, and $x_i \in \mathbb{R}^3$31 FPS rendering at $x_i \in \mathbb{R}^3$32, whereas PNeRFLoc reports $x_i \in \mathbb{R}^3$33 hour, $x_i \in \mathbb{R}^3$34 MB, and $x_i \in \mathbb{R}^3$35 FPS. On Replica novel-view rendering, average PSNR/SSIM/LPIPS across eight scenes are $x_i \in \mathbb{R}^3$36 for SplatLoc versus $x_i \in \mathbb{R}^3$37 for PNeRFLoc. For localization, SplatLoc achieves best or comparable results in $x_i \in \mathbb{R}^3$38 scenes of 12-Scenes with $x_i \in \mathbb{R}^3$39 cm and $x_i \in \mathbb{R}^3$40 median errors, and on Replica it outperforms PNeRFLoc scene by scene, for example $x_i \in \mathbb{R}^3$41 versus $x_i \in \mathbb{R}^3$42 on Room0 and $x_i \in \mathbb{R}^3$43 versus $x_i \in \mathbb{R}^3$44 on Office3 (Zhai et al., 2024).

The limitations of the two systems are partly shared and partly distinct. SplatLoc requires depth or a sparse point cloud to initialize splats and does not handle purely monocular reconstruction without priors; the authors suggest monocular depth such as DepthAnything as future support. Large outdoor scenes challenge memory and scale because they require many splats, and a hierarchical splat representation is suggested as a remedy. As with most feature-based pipelines, performance can degrade under extreme lighting changes, textureless regions, heavy dynamics, or severe occlusions, and localization depends on retrieval quality and descriptor distinctiveness. In SplatHLoc specifically, sparsely observed regions can produce rendering artifacts such as floaters or misfit geometry, repetitive structures can still induce incorrect alignments, and reduced training coverage lowers rendering fidelity, although nearest-view synthesis partially compensates for this in Cambridge (Zhai et al., 2024, Tao et al., 31 Mar 2026).

The future directions identified for SplatHLoc extend the splat-based hierarchical program rather than departing from it. Proposed directions include uncertainty-aware matching and splat covariances for weighting coarse matches, joint learning of features and poses by integrating the refinement loop into training, multi-scale or hierarchical splats, improved Gaussian initialization via large-scale 3D reconstruction foundation models, and block-wise partitioning for very large maps. Taken together with the earlier SplatLoc formulation, these directions suggest a broader research trajectory: replacing the sparse, static 3D point map in hierarchical localization with a compact, renderable, feature-aware Gaussian scene model that can support retrieval, correspondence generation, pose recovery, and real-time rendering within a single representation (Tao et al., 31 Mar 2026).