PiLoT v2: Pixel-to-Orthogonal Map Alignment for Free-view UAV Geo-localization

Abstract: Real-time, drift-free UAV geo-localization is essential for autonomous missions in GNSS-denied environments. The pioneering system, PiLoT, achieves high precision via Neural Pixel-to-3D Registration, aligning UAV video streams with a single rendered reference view from 3D meshes. However, its reliance on heavy 3D meshes incurs massive storage overheads, complex map acquisition, and significant computational rendering costs, severely hindering deployment on embedded platforms. To address these bottlenecks, we propose PiLoT v2, a lightweight yet robust evolution that shifts the paradigm to direct pixel-to-orthogonal map registration for free-view UAV geo-localization. By leveraging True Digital Orthophoto Maps (TDOMs) and Digital Surface Models (DSMs) as the reference substrate, PiLoT v2 replaces GPU-intensive 3D rendering with a highly efficient, CPU-friendly map cropping operation. To bridge the severe geometric discrepancy between these 2.5D orthogonal crops and free-view oblique UAV imagery, we train a cross-view feature registration network using a novel, large-scale geometrically annotated dataset. Furthermore, we integrate onboard sensor prior--specifically gravity direction and single-point laser rang--directly into the pose optimization manifold to enhance robustness against cross-view visual degradation. Experimental results demonstrate that PiLoT v2 achieves performance comparable to, or even exceeding, its Pixel-to-3D predecessor, while offering drastically lower storage and computational costs.

• The paper presents a pixel-to-orthogonal map alignment technique that leverages 2.5D geo-anchors for state-of-the-art UAV geo-localization performance.
• It employs a MobileOne-UNet backbone and multi-hypothesis Levenberg-Marquardt optimization to ensure precise cross-view feature matching and pose refinement.
• Experimental results demonstrate high recall and real-time performance across diverse weather, lighting, and pitch conditions on both real and synthetic benchmarks.

PiLoT v2: Pixel-to-Orthogonal Map Alignment for Free-view UAV Geo-localization

Introduction

PiLoT v2 addresses the critical challenge of robust and precise geo-localization for UAVs using image-based input in GNSS-denied settings. This work builds upon the PiLoT framework and introduces a highly efficient, real-time, and accurate solution for localizing UAV images against orthogonal maps (TDOM/DSM) at city scale. Unlike approaches dependent on online 3D mesh rendering or handcrafted image similarity, PiLoT v2 deploys a learnable pipeline that performs pixel-to-orthogonal map alignment with explicit 2.5D geo-anchors, sensor-prior fusion, and robust multi-hypothesis pose refinement.

Methodology

Reference Crop Generation and Geo-anchor Lifting

The reference cropping module establishes a pixel-aligned 2.5D representation by warping TDOM (True-Orthophoto) and DSM (Digital Surface Model) crops to match the current query viewpoint. Utilizing a lightweight homography-driven operation, it estimates the camera frustum’s map footprint and generates a synchronized alignment of appearance, elevation, and map coordinates (Figure 1).

Figure 1: Illustration of TDOM/DSM reference cropping, with all map layers warped into a pixel-aligned 2.5D crop for subsequent pose refinement.

Each valid reference pixel is lifted to a 3D geo-anchor, used as correspondence candidates in pose optimization. The DSM is only used for constructing metric anchors; it is never input directly to the network.

Cross-View Feature Extraction and Matching

The core backbone is a shared MobileOne-UNet architecture that processes both the query and the TDOM crop, generating a dense, L2-normalized feature pyramid and a confidence map per level. Descriptor alignment is enforced by a geometric loss applied throughout an unrolled feature-metric optimization, ensuring viewpoint and modality robustness across oblique and orthographic imagery.

Multi-Hypothesis LM Pose Refinement

Pose estimation is formulated as a coarse-to-fine LM optimization seeded by a dense grid of translation and yaw hypotheses. This initialization drastically enlarges the convergence basin, critical for robustness against significant prior errors. The optimization fuses dense visual feature-metric terms with range and gravity sensor priors, with all contributions balanced within the LM normal equation using adaptive gates.

Figure 2: Visualization of TDOM-to-query refinement, with arrow fields showing the update direction of dense geo-anchor projections under learned cross-view feature-metric residuals.

Asynchronous Runtime Architecture

PiLoT v2 organizes localization into two asynchronous threads: map preparation for reference crop generation and pose-localization for online optimization. This design enables real-time throughput by avoiding pipeline stalls due to map-data synchronization and maximizes efficiency by deploying a fused CUDA kernel for the dense residual computation during optimization.

Experimental Analysis

Real-world and Synthetic Trajectory Evaluation

Evaluation was performed on real-world UAVD4L-2yr, diverse SynthCity-6 synthetic, and UAVScenes public benchmarks (Figure 3, Figure 4, Figure 5). PiLoT v2 achieves robust, state-of-the-art trajectory accuracy under challenging scenarios, notably handling extreme appearance and long-term variations.

Figure 3: Trajectory estimation results on challenging real-world UAVD4L-2yr sequences, demonstrating robustness under severe day/night changes.

Figure 4: Trajectory estimation results exhibiting stable performance across diverse synthetic scenes and atmospheric conditions in SynthCity-6.

Figure 5: Strong generalization capability demonstrated on two unseen scenes from the UAVScenes benchmark dataset.

Weather, Illumination, and Pitch Robustness

Empirical analysis shows that PiLoT v2 maintains high recall (R@1/3/5/10) across all weather and lighting conditions in SynthCity-6, with especially strong performance under degraded, low-visibility conditions where competing methods degrade rapidly. Across both narrow nadir and wide oblique pitch distributions, PiLoT v2 consistently outperforms crop-based approaches and is competitive with 3D mesh-based render-comparison pipelines.

Multi-Hypothesis Ablation

The inclusion of joint (translation, yaw) multi-hypothesis sampling substantially improves recall and median error metrics, eliminating failure cases due to poor pose priors. Ablation experiments confirm that addressing both yaw and translation drift is critical for reliability in real-world deployment.

Runtime and Implementation

Online pose estimation per frame attains a total time of approximately 60 ms (excluding offline map build), with the majority spent in fused optimization. Map and sensor-prior terms are managed with adaptive weighting, ensuring stable performance even in the presence of outlier measurements. Feature-metric terms benefit from optimized GPU kernels, while irregular priors incur higher computation due to necessary coordinate conversions and DSM surface access.

Implications and Future Directions

PiLoT v2 demonstrates that efficient, learnable pixel-to-map registration can approach or exceed the performance of mesh-render-based pipelines, with a fraction of the computational cost and minimal pre-deployment overhead. The 2.5D crop approach bypasses the need for full 3D scene models at inference, while multi-sensor priors and cross-view feature learning ensure resilience to viewpoint, appearance, and sensor drift. These properties make PiLoT v2 suitable for global-scale, real-time UAV navigation in GNSS-denied, adverse, or dynamically changing environments.

Potential extensions include integration with dynamic scene understanding, explicit uncertainty modeling for sensor priors, and broader cross-domain generalization to ground and low-altitude agents. The approach also invites investigation into end-to-end flow unrolling and learned sensor weighting strategies, further automating adaptation to diverse operational regimes.

Conclusion

PiLoT v2 establishes a new standard for UAV geo-localization by jointly leveraging 2.5D pixel-to-map alignment, deep cross-view feature metric learning, and robust multi-hypothesis LM optimization with adaptive sensor fusion. Extensive benchmarks in real, synthetic, and public settings validate its strong localization accuracy, operational resilience, and generalizability. This framework provides a compelling foundation for scalable and reliable image-based navigation in large-scale, GPS-denied UAV applications.