Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and A Large-scale Benchmark

Abstract: Fine-grained RGBT image semantic segmentation is crucial for all-weather unmanned aerial vehicle (UAV) scene understanding. However, UAV RGBT semantic segmentation faces two coupled challenges: cross-modal spatial misalignment caused by sensor parallax and platform vibration, and severe semantic confusion among fine-grained ground objects under top-down aerial views. To address these issues, we propose a Graph-based Semantic Calibration Network (GSCNet) for unaligned UAV RGBT image semantic segmentation. Specifically, we design a Feature Decoupling and Alignment Module (FDAM) that decouples each modality into shared structural and private perceptual components and performs deformable alignment in the shared subspace, enabling robust spatial correction with reduced modality appearance interference. Moreover, we propose a Semantic Graph Calibration Module (SGCM) that explicitly encodes the hierarchical taxonomy and co-occurrence regularities among ground-object categories in UAV scenes into a structured category graph, and incorporates these priors into graph-attention reasoning to calibrate predictions of visually similar and rare categories.In addition, we construct the Unaligned RGB-Thermal Fine-grained (URTF) benchmark, to the best of our knowledge, the largest and most fine-grained benchmark for unaligned UAV RGBT image semantic segmentation, containing over 25,000 image pairs across 61 categories with realistic cross-modal misalignment. Extensive experiments on URTF demonstrate that GSCNet significantly outperforms state-of-the-art methods, with notable gains on fine-grained categories. The dataset is available at https://github.com/mmic-lcl/Datasets-and-benchmark-code.

• The paper introduces GSCNet that integrates feature decoupling, deformable alignment, and semantic graph calibration for improved accuracy in unaligned UAV RGBT segmentation.
• It employs FDAM to decouple features into structural and perceptual components, using illumination-aware alignment for robust fusion under varying light conditions.
• SGCM leverages a structured category graph to resolve semantic confusion, yielding a 10.99% mIoU gain over RGB-only baselines and significant improvements in tail-class recognition.

Graph-Based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation

Motivation and Technical Challenges

Unmanned aerial vehicle (UAV) RGBT scene segmentation presents unique technical barriers, chiefly cross-modal spatial misalignment and fine-grained semantic confusion. Sensor parallax, platform vibration, and view-dependent distortions result in spatially varying, object-dependent offsets between RGB and thermal imagery (Figure 1).

Figure 1: Cross-modal boundary misalignment in RGB-T UAV imaging. Blue contours denote RGB object boundaries; red contours show thermal boundaries. Spatial offsets are widespread across diverse object categories.

Such misalignment precludes naive pixel-wise fusion and impairs segmentation accuracy, notably at object boundaries and for small targets. Fine-grained semantic confusion is amplified under aerial perspectives, where objects like poles, streetlights, and traffic lights exhibit limited pixels and highly similar appearances in both modalities, resulting in persistent misclassification (Figure 2). Additionally, UAV-captured urban environments naturally exhibit long-tailed category distributions, rendering tail-class recognition unstable.

Figure 2: Semantic confusion matrix for pole, streetlight, and traffic light. Category confusion is exacerbated by visual similarity and small pixel coverage.

Methodology

GSCNet Architecture Overview

GSCNet is a unified spatial-semantic framework employing two core modules: FDAM for robust feature-level spatial correction, and SGCM for category-level semantic calibration in the presence of hierarchical and co-occurrence priors (Figure 3).

Figure 3: GSCNet processes RGB and thermal inputs via modality-specific MiT-B4 encoders. FDAM aligns features, and SGCM calibrates predictions by graph-attention over a structured category graph.

Feature Decoupling and Alignment Module (FDAM)

FDAM implements a decouple-then-align paradigm, grounded on the observation that object geometry is more invariant across modalities than appearance. Each modality’s features are decomposed into shared structural and private perceptual components using Asymmetric Feature Decoupling (AFD). Deformable alignment is then estimated in this shared subspace, leveraging illumination-aware anchor selection to adaptively reference RGB or thermal structure depending on scene context (Figure 4).

Figure 4: FDAM separates features into structural and perceptual branches; IAA performs deformable alignment with illumination-adaptive anchor selection.

A patch-level contrastive alignment loss, orthogonality regularization, and auxiliary segmentation loss jointly optimize the shared/structural representation. Illumination-aware alignment (IAA) uses a global weight $\lambda$ to select the more reliable anchor modality per image, facilitating robust cross-modal spatial fusion both in daytime (RGB-dominant) and nighttime (thermal-dominant) conditions.

Semantic Graph Calibration Module (SGCM)

SGCM models inter-category relations via a structured category graph $\mathcal{G}$ whose adjacency matrix integrates hierarchical similarity, contextual co-occurrence, and a learnable residual to adapt to dataset-specific patterns. Node features are initialized per-image by class-specific soft attention pooling over fused features and logits. Graph reasoning is performed via a prior-biased GAT, injecting adjacency priors as additive biases in attention computation, allowing rare and visually ambiguous categories to borrow discriminative context from semantically related nodes (Figure 5, Figure 6).

Figure 5: SGCM schematic: base predictions suffer from semantic confusion and tail-class errors; prior-enhanced graph attention corrects both error types.

Figure 6: Visualization of SGCM static prior matrices. Hierarchical similarity shows taxonomic block structure; co-occurrence captures scene-level contextual affinity.

URTF Benchmark

URTF represents the first large-scale benchmark for UAV RGBT semantic segmentation in the unaligned setting, comprising 25,519 RGB-T image pairs across 61 fine-grained categories. The data acquisition protocol preserves native sensor-level offsets, avoiding manual or algorithmic registration to maintain realistic fusion challenges.

URTF exhibits a three-level taxonomy and severe long-tailed distribution: five head categories comprise over 80% of all pixels, while 16 tail categories occupy less than 0.01% each (Figure 7, Figure 8). Scene diversity is achieved via real captures (urban, suburban, farmland, water) and synthetic samples (AirSim, CARLA), covering wide variation in illumination and adverse weather conditions (Figure 9, Figure 10).

Figure 9: Annotation examples across fog and low-light regimes; modality for annotation reference is condition-dependent.

Figure 10: Key scenes captured at altitudes of 50–300\,m for comprehensive coverage.

Figure 7: Hierarchical taxonomy: top-level groups, intermediate subclasses, leaf classes.

Figure 8: Long-tailed pixel distribution for all 61 semantic categories.

Empirical Evaluation

Quantitative Results

GSCNet achieves state-of-the-art performance on URTF, with 71.04% mean IoU and 60.17% IoU for the Tail-16 rarest categories. Compared to the strongest RGB-only baseline (SegFormer), GSCNet improves by 10.99% mIoU. Relative to AMDANet and MambaSeg, GSCNet yields clear gains in both mean and tail-class IoU, highlighting the necessity of both explicit spatial alignment and semantic graph calibration.

Qualitative Analysis

Visual inspection reveals GSCNet’s superiority in modality-adaptive structural fusion and semantic context calibration, especially under challenging conditions such as fog and night (Figure 11). Competing methods suffer from ghosting artifacts, boundary fragmentation, and tail-class suppression; GSCNet remedies these through FDAM’s spatial correction and SGCM’s contextual reasoning.

Figure 11: Daytime and nighttime scene segmentation results; GSCNet provides sharper boundaries and greater accuracy for small targets.

Ablation Studies

FDAM and SGCM are orthogonal; their integration yields additive gains in both mIoU and Tail-16. The decouple-then-align strategy is validated, as raw cross-modal offset estimation is unreliable without prior structural separation. SGCM’s knowledge fusion (taxonomy, co-occurrence, learnable residual) collectively boosts rare-category performance, and per-image node initialization via soft attention provides resilience to ambiguous pixel-level predictions (Figure 12).

Figure 12: FDAM, SGCM, and full model ablation: highest accuracy and border consistency achieved with full GSCNet.

Sensitivity analyses show peak performance at decoupling weight 0.1 and graph fusion weight 0.85, balancing spatial precision and semantic context (Figure 13).

Figure 13: Performance sensitivity to decoupling and graph fusion weight; optimal hyperparameters maximize mIoU.

Practical and Theoretical Implications

GSCNet’s formulation demonstrates that spatial misalignment and semantic confusion in multimodal UAV perception are mutually reinforcing challenges best addressed by sequential structural disentanglement, deformable alignment, and category-level graph reasoning. SGCM exemplifies the integration of dataset-derived structural priors with data-driven adaptation, suggesting generalization avenues for hierarchical and relational calibration in other fine-grained recognition tasks.

URTF establishes a rigorous evaluation standard, exposing both the value and limitations of current RGB-T fusion designs, and providing a resource for further multimodal research.

Speculation and Future Directions

Future exploration should address model compression and real-time deployment constraints, such as backbone adaptation for UAV hardware and knowledge distillation schemes. Extensions to cross-resolution RGB-T fusion, dynamic modality augmentation, and open-set segmentation are natural next steps. SGCM’s framework could be generalized to domains with latent relational structure (e.g., medical imaging, remote sensing), and its modular prior-integration could facilitate continual learning in rapidly evolving category taxonomies.

Conclusion

GSCNet delivers robust segmentation on unaligned UAV RGBT imagery by synergizing feature decoupling, illumination-aware alignments, and prior-enhanced semantic graph calibration. Empirical results on URTF confirm notable advances in both mean and tail-category recognition. These architectures and benchmark resources collectively advance the technical frontier in all-weather UAV scene understanding and establish design principles for future multimodal semantic segmentation systems.