Lights Out: A Nighttime UAV Localization Framework Using Thermal Imagery and Semantic 3D Maps

Abstract: Reliable backup localization for unmanned aerial vehicles (UAVs) operating in GNSS-denied nighttime conditions remains an open challenge due to the severe modality gap between daytime RGB maps and nighttime thermal imagery. This work presents a semantic reprojection framework for map-relative nighttime UAV localization by aligning segmented thermal observations with a globally referenced, semantically labeled 3D map constructed from daytime RGB data. Rather than relying on appearance-based correspondence, localization is formulated in a shared semantic domain and solved via a symmetric bidirectional reprojection objective with confusion-aware weighting to improve robustness under segmentation uncertainty. The approach is evaluated offline across 6.5 km of nighttime, real-world UAV flight trajectories in urban and semi-structured environments. Relative to RTK GNSS ground truth, the system achieves a bias-corrected RMSE2D of 2.18 m and a median RMSE2D of 1.52 m. Results show that localization performance is strongly correlated with the availability of semantic edge evidence and that large-error events are spatially localized to semantically ambiguous areas rather than uniformly distributed. These findings indicate that semantic reprojection offers a promising pathway toward globally referenced nighttime UAV localization using thermal imagery alone.

• The paper introduces a novel semantic alignment framework that enables drift-free, globally referenced UAV localization by matching thermal imagery with daytime 3D maps.
• It employs cross-modal semantic segmentation and reprojection-based edge detection to overcome challenges posed by low illumination and feature ambiguity.
• Experimental results demonstrate meter-level accuracy and robustness, validating the approach for UAV navigation in GNSS-denied environments.

Semantic Reprojection for Nighttime UAV Localization Using Thermal Imagery and 3D Semantic Maps

Introduction

The paper "Lights Out: A Nighttime UAV Localization Framework Using Thermal Imagery and Semantic 3D Maps" (2604.26201) presents a semantic alignment-based approach to precise localization of UAVs (Unmanned Aerial Vehicles) under GNSS-denied nighttime conditions. The core innovation is the formulation of global, map-relative localization as a semantic reprojection problem, aligning thermal infrared (LWIR) observations captured at night with a globally referenced, semantically labeled 3D map constructed from daytime RGB observations. By operating within a shared semantic domain, the method effectively eliminates the dependency on photometric and texture-based features, which are unreliable under severe illumination changes, and enables drift-free, globally referenced localization without requiring LiDAR sensors at inference time.

Figure 1: Overview of the proposed nighttime localization pipeline, illustrating construction of a semantic 3D map from daytime RGB imagery and subsequent alignment of nighttime thermal observations for global localization.

Problem Setting and Motivation

Nighttime UAV navigation is challenging due to the fundamental modality shift between visible-spectrum mapping assets (typically RGB-based satellite or aerial maps) and the nighttime onboard sensing modality (thermal infrared). Standard VIO or visual-SLAM pipelines exploit photometric consistency and fail in settings where RGB texture breaks down. LiDAR-based registration is illumination-invariant, but SWaP constraints and cost complicate adoption for lightweight UAV platforms [storch_comparative_2025]. Purely thermal-inertial odometry, as well as recent thermal-SLAM approaches [xu_slam_2025, li_wti-slam_2025, jiang_thermal-inertial_2022, chen_eamt-slam_2024], address illumination issues but accumulate unbounded drift in the absence of independent global corrections. Appearance-based geo-localization using satellite imagery [xiao_long-range_2023] exhibits poor cross-view and cross-modal generalizability at low altitudes and in scenes lacking discriminative appearance cues.

Instead, this work reformulates the global localization problem as a semantic alignment task, exploiting structural scene priors robust to extreme appearance changes. Semantic segmentation offers stable, persistent cues for cross-modal alignment, and a geometric reprojection-based alignment enables pose estimation that is inherently invariant to absolute radiometric values.

System Pipeline

The proposed framework decomposes the localization process into four major stages:

1. Daytime RGB Semantic Map Construction: RGB imagery collected during surveyed daytime flights is semantically segmented using a DeepLabV3+ network with MiT-B5 encoder, trained on a unified aerial taxonomy spanning FLAIR, UAVid, VDD, and Semantic Drone datasets. Segmentation targets core, structurally persistent classes: buildings, surfaces, tree/low vegetation, water, and vehicles.
2. Cross-modal Thermal Semantic Segmentation: Synchronized RGB-thermal image pairs are used to distill semantic supervision from the RGB model ("teacher") to the thermal model ("student") via projective homography alignment. The resulting thermal segmentation model is deployed at night for domain-adapted inference without the requirement for nighttime manual annotation.
3. Semantic 3D Map Construction via Reprojection and Edge Pruning: Semantic labels are assigned to a dense SfM point cloud (RealityCapture) via multi-view reprojection and majority voting. To emphasize registration-salient structure, the map is pruned to extract only voxels at class boundaries, yielding a compact semantic 3D edge map.
4. Nighttime Map-based Localization via Symmetric Semantic Reprojection: During night flights, observed thermal frames are segmented and edge maps extracted. The current pose prior is refined by searching for the translation minimizing a symmetric semantic Chamfer loss, matching projected map class boundaries with observed segmentation edges, with confusion-aware marginalization to account for thermal segmentation ambiguity.

   Figure 2: Pipeline overview on the City dataset, showing the offline construction and labeling of the 3D semantic map and the online, reprojection-based registration during night flights.

Semantic Segmentation: Cross-Modal Training and Efficacy

The RGB semantic model achieves an average IoU of 83.1% across unified classes on daytime aerial datasets, while the thermal student network yields 58.3% mean IoU, with best performance on tree vegetation, impervious, and low vegetation. Building and water classes are moderately represented, while vehicles are challenging due to both intrinsic thermal ambiguity and dataset class imbalance. Importantly, the modal classes that dominate aerial thermal imagery (vegetation, impervious, buildings) are segmented sufficiently well to provide robust, edge-driven structural constraints.

Low-altitude flights (e.g., 50m AGL) expose a limitation: insufficient class diversity leads to degenerate alignment, motivating future work on altitude-adaptive segmentation and expansion of training data diversity.

Semantic 3D Map and Edge-Driven Pruning

Semantic labels are fused across views using majority voting per point. Edge-aware pruning—retaining voxels at inter-class boundaries—reduces the point cloud to ≈4% of its original cardinality (e.g., 64M → 3M for the City dataset), significantly enhancing computational efficiency without sacrificing registration-relevant skeletal structure.

Symmetric Semantic Alignment for GNSS-Denied Nighttime Pose Estimation

The core localization objective is a symmetric, confusion-marginalized semantic Chamfer loss on the $xy$ translation:

• Forward term: Encourages projected map points (from the 3D semantic edge map) to correspond to edges of the same (or confusion-marginalized) class in the segmented thermal edge map.
• Reverse term: Enforces that observed segmentation edges are explained by projected map structure.

Both terms are weighted by empirical class confusion probabilities from daytime aligned RGB-thermal data, making the loss robust to known domain-specific ambiguities and misclassifications in the thermal domain. The optimization uses coarse-to-fine grid search, leveraging strong priors from onboard VIO/IMU for attitude and altitude, and focuses only on refining translation.

Experimental Evaluation

The system is evaluated across 6.5 km of collective nighttime flight trajectories covering urban, semi-structured, and waterfront environments. Daytime RTK ground-truth is used for validation only (not at inference). Key performance metrics:

• Localization error: 2D RMSE of 2.18 m and median error of 1.52 m across 1373 frames. 68% of frames localize within 2 m; the large-error ($>5$ m) tail comprises 6% of estimates.
• Component ablations: The symmetric bidirectional loss consistently suppresses variance (by $\approx$1.5 m) and failure cases compared to forward- or reverse-only terms. Confusion-aware marginalization reduces mean error by 12.1% and standard deviation by 17.5%.
• Computational cost: Average runtime for pose correction is 0.95 s/frame on an RTX 4070.

Critically, error events are not uniformly distributed but are spatially localized to three regimes: (i) regions with low semantic edge density (homogeneous terrain), (ii) systematic inter-class confusion (e.g., ambiguous water/impervious shoreline zones), and (iii) label or map-environment mismatch (vegetation changes). The remaining flight path exhibits high accuracy and stability, strongly correlated with edge density and structural class diversity.

Figure 3: Pier trajectory overlay with error color-coding versus RTK ground truth. Failure cases (black) are spatially concentrated in three unstructured regions (P1–P3).

Qualitative Examples and Failure Analysis

Qualitative results show that, in regions with high semantic detail and orthogonal classes, the system achieves errors $<0.4$ m ($\mathrm{RMSE}_{2D}$). In contrast, flights over large, homogeneous surfaces (a single building or water segment) or ambiguous boundaries result in errors up to 11 m due to insufficient geometric constraints and class confusion.

Figure 4: Top 10% localization: strong semantic structure and edge density drive sub-meter accuracy.

Implications and Future Directions

The results validate that semantically structured reprojection can provide robust, drift-free global pose corrections using thermal imagery alone, without appearance-based correspondence or active sensors at runtime. This framework is directly applicable for robust navigation in GNSS/visual-denied regimes—critical for safety-critical UAV deployments (e.g., infrastructure inspection, emergency response).

Key implications:

• Semantic structure (not texture/photometry) is sufficient for robust correspondence under severe domain shifts (e.g., day/night, RGB/thermal).
• Failure cases are dominated by segmentation quality and scene structural observability, not by the core loss or optimization.
• Direct extension to other sensory domains or further incorporation of uncertainty (pixelwise segmentation confidence, probabilistic model outputs) is plausible.
• Onboard integration with high-frequency thermal odometry for fused correction, rather than offline analysis, is the next step.

Expanded training datasets (especially with low-altitude, night, and under-represented classes) and improved segmentation architectures are expected to immediately translate into reduced error rates. Wider thermal FOV and increased image resolution could further improve context and reduce ambiguities.

Conclusion

This work demonstrates that semantic reprojection in a globally-referenced 3D frame is a viable solution for real-time, map-relative UAV localization under nighttime, RGB-degraded conditions. The method achieves meter-level accuracy, shows robust suppression of large-error outliers via loss symmetrization and confusion-aware marginalization, and is limited primarily by segmentation fidelity and scene edge observability rather than fundamental geometric constraints. The approach provides a sound basis for future work in robust, redundant state estimation for UAVs in GPS- or visually-denied environments and paves the way for the integration of semantically structured global localization with thermal-inertial odometry for entirely appearance-independent flight autonomy.