PlanaReLoc: Planar-Based 6-DoF Relocalization

• The paper introduces a novel plane-centric formulation that replaces traditional 2D–3D point matching with region-based planar matching for robust 6-DoF relocalization.
• It employs a transformer-based cross-modal matcher with normal-aware positional encoding to align query and map planar primitives, enhancing matching precision.
• The method recovers camera pose using RANSAC and weighted least-squares, achieving improved performance metrics on indoor datasets like ScanNet and 12Scenes.

PlanaReLoc is a structure-based camera relocalization method that formulates query-to-map association around planar primitives rather than 2D–3D point correspondences. Introduced in “PlanaReLoc: Camera Relocalization in 3D Planar Primitives via Region-Based Structure Matching” (Ye et al., 21 Mar 2026), it targets lightweight 6-DoF relocalization in structured indoor environments using a 3D planar map that can remain entirely untextured. Its central premise is that walls, floors, tabletops, doors, cabinets, and related planar surfaces are both geometrically fundamental and practically compact, making them suitable as the basic entities for cross-modal matching between a query RGB image and a geometry-only map.

1. Problem setting and plane-centric formulation

PlanaReLoc addresses camera relocalization: given a query image $q$, estimate the camera pose relative to a known scene map as

$P := [R \mid t] \in \mathrm{SE}(3),$

with $R \in \mathrm{SO}(3)$ and $t \in \mathbb{R}^3$ (Ye et al., 21 Mar 2026). The method departs from the dominant point-centric paradigm. Instead of recovering 2D–3D point correspondences and solving a PnP-style problem, it establishes correspondences between query-side planar regions and map-side 3D planar primitives.

The motivation is explicitly geometric and representational. Planes are treated as canonical entities in projective geometry, but also as region-based representations that aggregate evidence over extended image support. This region-based character is intended to improve robustness in weak-texture indoor scenes, partial occlusion, and repeated small-scale detail, while also enabling relocalization from a map that does not store realistic texture or color (Ye et al., 21 Mar 2026). The method is therefore positioned against several alternative regimes: SfM point maps, image-to-render matching against textured maps, and image-to-point-cloud registration across modalities.

The target domain is structured indoor environments. The paper emphasizes that such scenes are dominated by planar surfaces and that a planar map is substantially lighter than dense textured meshes, dense point clouds, or large keypoint maps. A practical implication is that PlanaReLoc is designed for settings where memory footprint and map appearance requirements matter as much as relocalization accuracy.

2. Query and map primitives

The planar map is denoted

$\mathcal{M} = \{m_i\}_{i=1}^{N_m},$

where each map primitive $m_i$ carries plane parameters $\pi_i^m$ and a bounded spatial shape $s_i^m$ (Ye et al., 21 Mar 2026). Planes are parameterized as

$\pi := [n^\top, d]^\top,$

with $n \in \mathbb{R}^3$ the plane normal and $P := [R \mid t] \in \mathrm{SE}(3),$0 the offset. On the map side, the normal orientation is kept consistent with the original surface normal.

On the query side, the image is converted into a set of recovered planar primitives

$P := [R \mid t] \in \mathrm{SE}(3),$1

each with predicted metric plane parameters $P := [R \mid t] \in \mathrm{SE}(3),$2 and a binary 2D mask $P := [R \mid t] \in \mathrm{SE}(3),$3 (Ye et al., 21 Mar 2026). Query planes are not obtained from a semantically heavy detector; instead, the paper uses MoGe-2 for monocular metric geometry estimation followed by sequential RANSAC plane fitting. This front-end is described as purely geometric and plug-and-play.

Several implementation details delimit the primitive extraction regime. The input image is resized to $P := [R \mid t] \in \mathrm{SE}(3),$4; the feature map is $P := [R \mid t] \in \mathrm{SE}(3),$5; fitting operates on downsampled depth; the inlier threshold is a point-to-plane residual below 10 cm and a normal-similarity dot product above 0.9; and extraction stops after 16 planes or when the inlier count falls below 1% of pixels (Ye et al., 21 Mar 2026). The result is a compact query representation consisting of at most 16 planar regions.

The query embeddings are formed by pooling image features over each recovered segment mask. Specifically, the image is patchified by a pretrained encoder, masks are resized to feature-map resolution, and average pooling over each segment produces query-plane embeddings

$P := [R \mid t] \in \mathrm{SE}(3),$6

with embedding dimension $P := [R \mid t] \in \mathrm{SE}(3),$7 (Ye et al., 21 Mar 2026). On the map side, embeddings

$P := [R \mid t] \in \mathrm{SE}(3),$8

combine two PointNet-based components: an object encoder over centralized per-plane point clouds and a scene encoder that captures pose-aware spatial context in the whole map. Each primitive is represented with $P := [R \mid t] \in \mathrm{SE}(3),$9 sampled points, and the two embeddings are fused by a learnable $R \in \mathrm{SO}(3)$0-weighted sum (Ye et al., 21 Mar 2026).

This two-branch map encoding is structurally important. The object encoder captures local plane shape, while the scene encoder disambiguates repeated local geometry by encoding scene context. The paper’s interpretation is that neither local shape alone nor scene context alone is sufficient for reliable planar matching in indoor scenes with repeated walls or furniture panels.

3. Cross-modal planar matching

PlanaReLoc uses a transformer matcher to associate query-plane embeddings and map-plane embeddings inside a shared cross-modal embedding space (Ye et al., 21 Mar 2026). The architecture is a stack of $R \in \mathrm{SO}(3)$1 identical transformer layers, each with one self-attention unit and one cross-attention unit, and each unit uses 4 attention heads. The model is inspired more by assignment learning in the style of SuperGlue and LightGlue than by pure contrastive embedding objectives.

A key geometric design is the injection of relative normal information into self-attention. The paper writes the modified self-attention score as

$R \in \mathrm{SO}(3)$2

with a rotary positional encoding

$R \in \mathrm{SO}(3)$3

where each $R \in \mathrm{SO}(3)$4 is learnable (Ye et al., 21 Mar 2026). The operational role of this term is to expose relative plane-orientation cues directly inside attention.

The matcher predicts a soft assignment matrix

$R \in \mathrm{SO}(3)$5

with entries combining pairwise similarity and predicted matchability: $R \in \mathrm{SO}(3)$6 Here,

$R \in \mathrm{SO}(3)$7

and the per-primitive matchability is

$R \in \mathrm{SO}(3)$8

(Ye et al., 21 Mar 2026). At inference time, correspondences are accepted only if they pass a confidence threshold $R \in \mathrm{SO}(3)$9 and satisfy the Mutual Nearest Neighbor criterion.

Training labels are constructed from ground-truth camera pose by projecting map primitives into the query image and selecting, for each recovered query primitive, the map primitive with the highest projected mask IoU (Ye et al., 21 Mar 2026). Importantly, the supervision is not bipartite: one map primitive may supervise multiple query primitives, reflecting over-segmentation in the query due to occlusion, truncation, or imperfect plane extraction. Unmatchable query and map primitives are collected into $t \in \mathbb{R}^3$0 and $t \in \mathbb{R}^3$1 based on an IoU threshold $t \in \mathbb{R}^3$2.

The matching objective is

$t \in \mathbb{R}^3$3

and is applied at every transformer layer for deep supervision (Ye et al., 21 Mar 2026).

4. Pose recovery from matched planes

Once plane correspondences are established, PlanaReLoc solves pose directly from plane geometry. If the camera pose maps camera-space points to map-space points as $t \in \mathbb{R}^3$4, plane parameters transform by

$t \in \mathbb{R}^3$5

From this, the paper derives

$t \in \mathbb{R}^3$6

and

$t \in \mathbb{R}^3$7

Thus, matched normals constrain rotation, while offsets constrain translation once rotation is known (Ye et al., 21 Mar 2026).

Rotation is estimated robustly with RANSAC. Two non-parallel matched plane pairs form the minimal sample used to hypothesize $t \in \mathbb{R}^3$8; the best inlier set $t \in \mathbb{R}^3$9 is then used to recompute an initial rotation $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$0 via the Kabsch algorithm (Ye et al., 21 Mar 2026). Because monocular plane recovery introduces scale error into query plane offsets, translation is solved jointly with a global scale factor $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$1: $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$2 The weight $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$3 is proportional to the 2D segment size of query primitive $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$4, reflecting the assumption that larger recovered planes are more reliable (Ye et al., 21 Mar 2026).

The paper rewrites this as a weighted linear least-squares problem. For each inlier correspondence $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$5,

$\mathcal{M} = \{m_i\}_{i=1}^{N_m},$6

which yields

$\mathcal{M} = \{m_i\}_{i=1}^{N_m},$7

and

$\mathcal{M} = \{m_i\}_{i=1}^{N_m},$8

Translation requires at least three non-parallel correspondences, and the reported degeneracy rate is below 2% empirically (Ye et al., 21 Mar 2026). When degeneracy occurs, the method falls back to a heuristic coarse pose estimated from predicted correspondences.

5. Depth-based refinement and empirical performance

PlanaReLoc includes a post-estimation refinement stage that jointly optimizes pose and per-query-plane offset corrections against a rendered depth map from the planar map (Ye et al., 21 Mar 2026). The initial pose is denoted $\mathcal{M} = \{m_i\}_{i=1}^{N_m},$9, and the refined pose is

$m_i$0

where $m_i$1 is a relative transformation and $m_i$2 are per-plane offset seeds. For each query primitive $m_i$3, the warped depth segment is computed by

$m_i$4

and the per-primitive residual is

$m_i$5

The aggregate refinement objective is

$m_i$6

Optimization uses Adam for 200 iterations, with $m_i$7 parameterized as a differentiable 6D Lie algebra variable via LieTorch, learning rate $m_i$8 for $m_i$9, learning rate $\pi_i^m$0 for $\pi_i^m$1, and 4096 sampled pixels per iteration (Ye et al., 21 Mar 2026).

The reported relocalization results on ScanNet and 12Scenes summarize both the base solver and the effect of refinement.

Dataset	Variant	Key reported performance
ScanNet	Without refinement	Mean rot err 17.3, median rot err 3.9, mean trans err 0.65, median trans err 0.27, recalls 37.1 / 69.8 / 79.8, runtime 0.059854 s/query
ScanNet	Full PlanaReLoc	Mean rot err 17.2, median rot err 3.8, mean trans err 0.60, median trans err 0.20, recalls 48.5 / 73.1 / 81.8, runtime 0.544342 s/query
12Scenes	Without refinement	Mean rot err 4.8, mean trans err 0.28, recalls 34.9 / 66.7 / 79.9
12Scenes	Full PlanaReLoc	Mean rot err 4.7, mean trans err 0.19, recalls 50.6 / 70.8 / 80.6

The paper also reports explicit matching metrics. On ScanNet, PlanaReLoc reaches precision 67.6, recall 61.3, F-score 64.3, and AP 91.8. On 12Scenes, it reaches precision 63.9, recall 54.2, F-score 58.6, and AP 87.8 (Ye et al., 21 Mar 2026). These results are used to support the claim that planar primitives are effective for cross-modal structural matching between an RGB query and an untextured planar map.

Ablation studies indicate that several components are not incidental. Removing either the object encoder or the scene encoder degrades performance, as does removing the normal-based positional embedding. Robust estimation with RANSAC is reported as crucial, and explicit optimization of the global monocular scale improves pose accuracy (Ye et al., 21 Mar 2026). The default query front-end, MoGe-2 + RANSAC, also gives the best reported speed/accuracy tradeoff among tested plane-recovery alternatives.

6. Map compactness, comparative position, and limitations

A major practical feature of PlanaReLoc is map compactness. The appendix reports that simplified planar maps average 154.3 KiB, which is only 3.2% of the colored map size (Ye et al., 21 Mar 2026). This is central to the method’s deployment rationale: relocalization is performed without textured or colored maps, without pose priors, and without per-scene training.

This places PlanaReLoc in a distinct part of the localization design space. Unlike LaLaLoc, which localizes a single RGB panorama to a floor plan by learning a shared latent layout space and estimates a 2-DoF planar pose in previously unvisited indoor scenes (Howard-Jenkins et al., 2021), PlanaReLoc solves 6-DoF relocalization against a 3D planar map. Unlike UnLoc, which performs sequential floorplan localization in $\pi_i^m$2 using uncertain 1D floorplan-depth signatures and histogram filtering (Wüest et al., 14 Sep 2025), PlanaReLoc is not a floorplan-localization method and does not operate over occupancy-grid pose volumes. It is also distinct from PlaneRecTR++, which learns joint plane reconstruction and relative camera pose between two images rather than absolute query-to-map relocalization (Shi et al., 2023).

The paper identifies several limitations. The dominant bottleneck is monocular plane recovery: if the front-end produces poor plane segments or inaccurate plane geometry, matching and pose estimation can fail (Ye et al., 21 Mar 2026). Highly repetitive structures remain problematic; the appendix includes a case in which repeated planar layouts allow RANSAC to preserve an incorrect pose despite several correct matches. Performance also degrades in scenes with too few informative planes, in large multi-room environments with increasing structural ambiguity, and beyond the indoor structured regime for which the method was designed. The authors explicitly note that outdoor generalization is not established (Ye et al., 21 Mar 2026).

Within those bounds, PlanaReLoc’s main technical significance lies in its reformulation of relocalization as region-based structure matching over planar primitives. The method combines a query-side geometric plane recovery front-end, a map-side geometric embedding for untextured planes, a transformer assignment model with explicit matchability, a closed-form plane-based pose solver augmented by global scale compensation, and a depth-render refinement stage. This architecture suggests that, in structured indoor scenes, planar primitives can function not merely as auxiliary cues but as the primary entities for cross-modal relocalization (Ye et al., 21 Mar 2026).