Photo–Floor Plan Correspondence

• Photo–floor plan correspondence is a cross-modal challenge that aligns photographic images with 2D/3D architectural plans using geometric and semantic cues.
• It integrates modular pipelines and end-to-end architectures, employing techniques like structure-from-motion, edge detection, and contrastive losses to bridge modality gaps.
• Advanced methods, such as PALMS+ and DisCo-FLoc, quantitatively assess performance with metrics like RMSE and localization accuracy for applications in indoor navigation and augmented reality.

Photo–floor plan correspondence denotes a class of geometric and cross-modal computer vision problems in which photographs, typically captured from egocentric or exocentric viewpoints, are algorithmically matched or aligned with architectural floor plans. The objective may be camera localization within architectural drawings, pixel-wise correspondences across modalities, or the fusion of RGB imagery with 2D/3D schematic priors. This correspondence underpins numerous domains such as infrastructure-free indoor navigation, augmented reality, construction monitoring, and embodied navigation. Addressing this problem requires simultaneously resolving geometric alignment, viewpoint–modality gaps, and semantic ambiguities.

1. Foundational Problem Formulations and Dataset Advances

Photo–floor plan correspondence spans several technical formulations: pose estimation (recovering $(x,y,\theta)$ of the camera in the plan frame), semantically or geometrically dense correspondence (mapping photo pixels to plan coordinates), and registration/alignment of photo-derived evidence with architectural blueprints. The C3Po approach formalizes the dense correspondence task as a learned mapping $C:\Omega_p \to \Omega_f$, where $\Omega_p$ and $\Omega_f$ are pixel domains of the photo and floor plan, respectively. Pixel-level ground-truth is established by reconstructing a 3D point cloud via structure-from-motion, manually registering it to the 2D plan, and projecting visible 3D points into photo and plan coordinates, yielding 153 million correspondences over 597 scenes in the C3 dataset (Huang et al., 23 Nov 2025).

Dataset innovations such as C3 enable empirical benchmarking across diverse, real-world scenarios and modalities, exposing the limitations of prior art trained only on image-image pairs or lacking ground truth for geometric correspondence.

2. Modular and End-to-End Algorithmic Pipelines

Modern photo–floor plan correspondence pipelines often follow a modular structure comprising a geometric observation stage and a layout/matching backend. PALMS+ exemplifies this modularity by first reconstructing a scale-aligned 3D point cloud from posed RGB images using a monocular depth foundation model (Depth Pro), correcting per-view and global scale ambiguities via RANSAC ground-plane fitting. The resulting 3D point cloud is projected to 2D via selection of points near the canonical camera height, rasterized, and processed to extract line segments representing observed layout via edge detection and Hough transforms (Cheng et al., 12 Nov 2025).

Alternative end-to-end architectures adopt learned fusion strategies. FloNa utilizes parallel EfficientNet-B0 backbones for image and plan feature extraction, fuses appearance and geometry using a multi-headed attention transformer, and regresses camera pose via supervised MLP heads using explicit MSE losses for both pose and plan-based geodesic distance estimates (Li et al., 2024).

The methods can be instantiated in stationary (single-shot localization) or sequential (particle-filtered tracking) modes, depending on application.

3. Geometric and Semantic Cues for Cross-Modal Alignment

Addressing the modality gap between photos (with rich appearance and perspective) and floor plans (abstract, schematic, and top-down) is foundational. Alignment methods leverage geometric cues (edge, line, wall structure; depth or layout prediction from images), semantic cues (doors as robust markers), and cross-modal feature fusion.

For example, in PALMS+, the observed wall layout bitmap is aligned to the floor plan via convolutional matching using Bayesian inference over $(x,\theta)$: a convolved kernel combines the rotated observed wall bitmap and an empty-space mask encoding ray visibility, with the resulting likelihood heatmap serving as a proxy for $p(x,\theta|I,F)$ (Cheng et al., 12 Nov 2025). In “Exploiting 2D Floorplan for Building-scale Panorama RGBD Alignment”, geometric evidence (point-evidence and free-space masks from RGBD panoramas) is combined with semantic door cues to define unaries in a Markov Random Field, and a coverage potential is imposed to avoid degenerate scan stacking (Wijmans et al., 2016).

DisCo-FLoc introduces a dual-level contrastive loss that operates over feature encodings of photo patches and floor plan snippets, explicitly enforcing both positional and orientation-level discrimination by generating hard negative samples via translation and rotation (Meng et al., 5 Jan 2026). This scheme removes ambiguities in repetitive or minimalist floor plans not addressable by geometry alone.

4. Probabilistic Models, Contrastive Learning, and Failure Modes

Probabilistic modeling is pervasively used to encode pose or correspondence uncertainty. DisCo-FLoc discretizes the pose space $(x,y,\theta)$, forming a dense inference tensor, and first applies a ray-regression predictor to generate candidate pose likelihoods via comparison of predicted depth rays with simulated rays at candidate poses. The top candidates are disambiguated through visual–geometric contrastive losses—composed over position-level and orientation-level negatives—leveraging backbone feature encodings to maximize alignment accuracy in ambiguous architectural contexts (Meng et al., 5 Jan 2026).

C3Po, in contrast, directly regresses per-pixel 3D coordinates in the floor-plan coordinate frame, optimizing a multi-scale Huber loss with an uncertainty-aware confidence regulation term. Qualitative analysis reveals strong performance in structurally unique scenes but failure in minimal-context images and highly symmetric plans, where global context is insufficient (Huang et al., 23 Nov 2025).

Failure modes across approaches frequently include structurally homogeneous spaces (e.g., box-shaped rooms, long corridors) manifesting in multimodal, ambiguous likelihood maps; scale drift or glass-induced spurious features; and plan–scene discrepancies due to unmodeled furniture or outdated documentation.

5. Evaluation Protocols, Quantitative Benchmarks, and Systemic Limitations

Evaluation metrics for photo–floor plan correspondence are specialized:

• Localization accuracy at defined thresholds (e.g., $1\,\text{m}/30^\circ$) (Cheng et al., 12 Nov 2025, Meng et al., 5 Jan 2026)
• Root-mean-square error (RMSE) and percentage of correct keypoints (PCK) (Huang et al., 23 Nov 2025)
• Success rate and SPL for navigation (Li et al., 2024)
• Placement error rates verified by manual inspection (Wijmans et al., 2016)
• Room/corner recall and IoU against point-cloud and LiDAR-based baselines (Solarte et al., 2021)

PALMS+ attains 30.4% accuracy @1m/30° in large campus buildings (outperforming image and LiDAR baselines) and achieves an RMSE of 2.2m for sequential tracking (Cheng et al., 12 Nov 2025). DisCo-FLoc realizes a Recall@1m of 56.7% on Gibson(f), a ~9% improvement over predecessors (Meng et al., 5 Jan 2026). C3Po achieves a 34% RMSE reduction over prior art, although global context and symmetry remain unresolved challenges (Huang et al., 23 Nov 2025). 360-DFPE, fusing panoptic layout and monocular SLAM, outperforms full-pointcloud and LiDAR-methods in room-and-corner recall on large-scale datasets (Solarte et al., 2021).

End-to-end system limitations include reliance on supervised point-cloud coverage for ground-truth, the inability to express posterior uncertainty in a principled manner, sensitivity to plan inaccuracies or outdated documentation, and the open challenge of bridging severe cross-modal domain shifts.

6. Applications, Generalization, and Prospects

Photo–floor plan correspondence methods now enable plug-and-play, zero-shot indoor localization, infrastructure-free AR, rapid mapping in GPS-denied environments, and embodied navigation with online plan guidance. Due to architectural modularity, state-of-the-art methods such as PALMS+ act as composable toolkits accepting improved monocular depth estimators or enhanced map-matching backends without retraining (Cheng et al., 12 Nov 2025). MRF-based alignment generalizes to other imaging modalities (e.g., LiDAR, monocular panoramas, time-lapse construction scans), multi-level architectural plans, and diverse building typologies (Wijmans et al., 2016).

Future extensions identified in the literature include explicit probabilistic correspondence (normalizing flows, diffusion models), joint pose–map estimation objectives, cross-modal transformers, generative plan–image synthesis, and robust handling of ambiguous or incomplete evidence (Huang et al., 23 Nov 2025). Continuing advances in large-scale datasets (C3), contrastive training paradigms, and depth foundation models define the trajectory of this rapidly advancing domain.