Graph-based Semantic Calibration Network

Updated 4 July 2026

The paper introduces a dual-branch transformer that integrates feature decoupling and semantic graph calibration to correct cross-modal misalignments and fine-grained class confusions in UAV imagery.
FDAM effectively decomposes features into shared structural and modality-specific components, using deformable alignment and illumination-aware blending to address spatial misalignment.
SGCM leverages hierarchical taxonomy and co-occurrence priors within a graph attention framework to refine segmentation logits, yielding significant gains, especially for rare classes.

Searching arXiv for the specific GSCNet paper and closely related graph-based semantic calibration work. Graph-based Semantic Calibration Network (GSCNet) is a dual-branch transformer-based architecture for unaligned UAV RGB–thermal semantic segmentation in fine-grained aerial scenes. It was introduced for the setting in which RGB and thermal image pairs exhibit realistic cross-modal spatial misalignment while also requiring discrimination among 61 semantically confusable ground-object categories. In that formulation, GSCNet combines a Feature Decoupling and Alignment Module (FDAM) with a Semantic Graph Calibration Module (SGCM), and is presented together with the Unaligned RGB–Thermal Fine-grained (URTF) benchmark, a large-scale dataset constructed to preserve realistic misalignment and long-tailed class structure (Fan et al., 29 Apr 2026).

1. Problem setting and motivating difficulties

GSCNet is defined for semantic segmentation from unaligned RGB–thermal image pairs captured by unmanned aerial vehicles. The problem formulation emphasizes two coupled challenges: cross-modal spatial misalignment and fine-grained semantic confusion under top-down views. The first arises because RGB and thermal cameras have non-coincident centers, different optics, and are affected by platform vibration and view-dependent parallax, producing spatially varying, object-dependent offsets that cannot be corrected by a single global transformation. The second arises because the task involves 61 categories, many of which are visually similar, small, and severely imbalanced in pixel frequency (Fan et al., 29 Apr 2026).

The long-tailed structure of URTF is central to this formulation. Five head classes cover approximately 80.7% of pixels, whereas 16 tail classes each occupy less than 0.01% of pixels. Under aerial imaging conditions, categories such as pole, streetlight, and traffic light, as well as multiple vehicle subtypes and ground-surface categories such as road, impervious, and parking lot, can have similar RGB and thermal appearance. This makes flat pixel-wise classification unstable, especially when local evidence is further degraded by modal misalignment (Fan et al., 29 Apr 2026).

GSCNet addresses these issues through what the paper characterizes as a spatial–semantic two-stage strategy. FDAM operates on feature alignment by decomposing each modality into shared structural and private perceptual components and then performing deformable alignment in the shared space. SGCM operates on semantic reasoning by encoding hierarchical taxonomy and co-occurrence regularities into a structured category graph and using graph-attention reasoning to calibrate predictions of visually similar and rare categories (Fan et al., 29 Apr 2026).

This suggests that the defining feature of GSCNet is not graph processing alone, but the coupling of geometric correction and category-level relational correction within a single segmentation framework.

2. Architectural composition

The overall network uses dual MiT-B4 SegFormer encoders, one for RGB and one for thermal, with independent parameters. Each encoder outputs four-stage feature maps,

$F_R^{(i)}, F_T^{(i)}, \quad i=1,\dots,4,\quad C_i \in \{64,128,320,512\}.$

At each stage, FDAM is applied to the two modality features. The aligned and fused stage-wise representations $\hat{F}^{(i)}$ are then passed to the standard SegFormer MLP decoder to obtain a fused representation $\mathbf{F}_{\mathrm{fuse}}$ and base logits $\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ , where $K=61$ . SGCM then takes $\mathbf{F}_{\mathrm{fuse}}$ and $\mathbf{L}_0$ , produces graph-calibrated logits $\mathbf{L}_g$ , and fuses them with $\mathbf{L}_0$ to produce the final segmentation (Fan et al., 29 Apr 2026).

The architecture is therefore organized around two complementary operators. FDAM is responsible for cross-modal reconciliation before decoding, whereas SGCM is responsible for category-level correction after the decoder has already produced base spatial logits. A plausible implication is that the network treats alignment errors and semantic errors as partially separable phenomena: one is addressed in the latent spatial representation, the other in the class-relation space.

The design also places GSCNet within a broader trajectory of graph-based semantic reasoning in dense prediction. In weakly supervised semantic segmentation, graph reasoning has likewise been used to model inter-class dependencies and contextual priors, although in a different supervision regime and task setting (Zhang et al., 2023). By contrast, GSCNet instantiates graph reasoning in a multimodal aerial segmentation system in which category priors are directly fused back into pixel logits (Fan et al., 29 Apr 2026).

3. Feature Decoupling and Alignment Module

FDAM consists of Asymmetric Feature Decoupling (AFD) and Illumination-Aware Alignment (IAA). For each stage $i$ , AFD decomposes modality features into shared structural features and private perceptual features:

$\hat{F}^{(i)}$ 0

$\hat{F}^{(i)}$ 1

Here, $\hat{F}^{(i)}$ 2 is a shared two-layer Conv–BN–ReLU encoder used for both modalities, and $\hat{F}^{(i)}$ 3, $\hat{F}^{(i)}$ 4 are lighter modality-specific one-layer encoders (Fan et al., 29 Apr 2026).

To enforce this decomposition, FDAM uses three losses. First, a patch-based contrastive alignment loss $\hat{F}^{(i)}$ 5 is applied to the shared features after 8×8 average pooling and $\hat{F}^{(i)}$ 6 normalization, using an InfoNCE-style objective with temperature $\hat{F}^{(i)}$ 7:

$\hat{F}^{(i)}$ 8

Second, an orthogonality loss $\hat{F}^{(i)}$ 9 penalizes squared mean cosine similarity between shared and private components to prevent collapse into a common subspace. Third, an auxiliary semantic loss $\mathbf{F}_{\mathrm{fuse}}$ 0 supervises shared features with a lightweight segmentation head and downsampled labels:

$\mathbf{F}_{\mathrm{fuse}}$ 1

Over four stages, these combine into

$\mathbf{F}_{\mathrm{fuse}}$ 2

with $\mathbf{F}_{\mathrm{fuse}}$ 3, $\mathbf{F}_{\mathrm{fuse}}$ 4, and $\mathbf{F}_{\mathrm{fuse}}$ 5 (Fan et al., 29 Apr 2026).

IAA then performs deformable alignment in the shared structural space. For each stage, two offset predictors estimate forward and backward deformable offsets and modulation masks:

$\mathbf{F}_{\mathrm{fuse}}$ 6

$\mathbf{F}_{\mathrm{fuse}}$ 7

Using DCNv2-style deformable convolution, the network obtains warped shared features in each other’s reference frame:

$\mathbf{F}_{\mathrm{fuse}}$ 8

$\mathbf{F}_{\mathrm{fuse}}$ 9

The illumination-aware component introduces a scalar $\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 0 derived from the RGB image:

$\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 1

with a two-layer MLP of hidden size 16. The aligned features are then blended as

$\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 2

$\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 3

When $\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 4, RGB acts as the more reliable reference; when $\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 5, thermal acts as the more reliable reference (Fan et al., 29 Apr 2026).

The same offsets are also applied to private features, with modality-specific DCN weights, to maintain geometric consistency across shared and private branches. After alignment, stage-wise fusion is defined as

$\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 6

This formulation makes explicit that alignment is learned from shared structural space but preserved across the full fused representation (Fan et al., 29 Apr 2026).

4. Semantic Graph Calibration Module

SGCM is the graph-based semantic component of GSCNet. It defines a category graph

$\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 7

where each node corresponds to a semantic category. The prior adjacency is built from two sources: a hierarchical similarity prior $\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 8 derived from a three-level taxonomy with six top-level groups, and a co-occurrence prior $\mathbf{L}_0 \in \mathbb{R}^{B \times K \times H \times W}$ 9 computed from training-image statistics (Fan et al., 29 Apr 2026).

The hierarchical prior is defined by

$K=61$ 0

where $K=61$ 1 is the tree path distance and $K=61$ 2. The co-occurrence prior is

$K=61$ 3

The two are combined as

$K=61$ 4

symmetrized by

$K=61$ 5

and normalized to

$K=61$ 6

A learnable residual adjacency $K=61$ 7 is then added:

$K=61$ 8

followed by a final symmetric normalization to yield $K=61$ 9. To constrain deviation from the priors, SGCM adds an $\mathbf{F}_{\mathrm{fuse}}$ 0 penalty

$\mathbf{F}_{\mathrm{fuse}}$ 1

This construction encodes taxonomy, contextual compatibility, and data-driven residual correction in a single graph (Fan et al., 29 Apr 2026).

Image-specific node features are derived from the current fused feature map and base logits. With

$\mathbf{F}_{\mathrm{fuse}}$ 2

the node embedding for class $\mathbf{F}_{\mathrm{fuse}}$ 3 in image $\mathbf{F}_{\mathrm{fuse}}$ 4 is

$\mathbf{F}_{\mathrm{fuse}}$ 5

The use of soft attention means that ambiguous pixels may contribute to multiple class nodes rather than being committed to a hard assignment at an early stage (Fan et al., 29 Apr 2026).

Reasoning over these nodes is performed by a prior-biased Graph Attention Network. Attention logits are computed as

$\mathbf{F}_{\mathrm{fuse}}$ 6

and the prior adjacency enters as a log-bias:

$\mathbf{F}_{\mathrm{fuse}}$ 7

The module uses a two-layer, four-head GAT. The refined node features

$\mathbf{F}_{\mathrm{fuse}}$ 8

are projected back to the spatial domain through

$\mathbf{F}_{\mathrm{fuse}}$ 9

The final logits are obtained by residual fusion:

$\mathbf{L}_0$ 0

with $\mathbf{L}_0$ 1 (Fan et al., 29 Apr 2026).

In this formulation, semantic calibration refers specifically to class-level relational correction of base logits. That usage differs from graph calibration work in node classification, where calibration usually means confidence correction through nodewise temperatures rather than logit refinement in pixel space (Tang et al., 2023). The overlap lies in the use of graph structure to encode semantic or similarity priors; the operational target differs.

5. Training objectives and benchmark setting

The total loss is

$\mathbf{L}_0$ 2

with $\mathbf{L}_0$ 3 and $\mathbf{L}_0$ 4. The segmentation loss $\mathbf{L}_0$ 5 is OHEM cross-entropy on final logits with hard pixel threshold $\mathbf{L}_0$ 6 (Fan et al., 29 Apr 2026).

GSCNet is introduced together with URTF, the Unaligned RGB–Thermal Fine-grained benchmark. URTF contains 25,519 RGB–T image pairs at resolution 640 × 512, including 16,894 real UAV pairs and 8,625 synthetic pairs generated with AirSim and CARLA. The training split contains 20,393 pairs, and the validation split contains 5,126 real pairs. The benchmark includes 14,844 daytime, 6,900 foggy, and 3,775 low-light image pairs (Fan et al., 29 Apr 2026).

The benchmark is constructed to preserve realistic misalignment. Images are cropped or resized so that both modalities are 640 × 512, but no registration is applied: there is no global calibration and no local warping in the data itself. Misalignment is attributed to physical baseline, different focal lengths, parallax from scene depth, and UAV motion or vibration. Annotations are drawn on whichever modality has clearer structure in each region, typically RGB in daytime and thermal in low-light settings. Consequently, labels are defined in the frame of the annotated modality, while the other modality remains misaligned (Fan et al., 29 Apr 2026).

URTF also formalizes the category taxonomy used by SGCM. It contains 61 semantic categories arranged in a three-level hierarchy with six top-level groups. The annotation process involved 30 professional annotators over approximately six months and more than 28k person-hours, with three rounds of expert inspection and correction, and approximately 99.992% of pixels labeled. Evaluation uses mIoU, mAcc, aAcc, Head-5 IoU, and Tail-16 IoU (Fan et al., 29 Apr 2026).

6. Empirical performance and ablation structure

On URTF validation, GSCNet reports 71.04% mIoU, 78.97% mAcc, 92.65% aAcc, 87.40% Head-5 IoU, and 60.17% Tail-16 IoU. The best RGB-only baseline reported is SegFormer (RGB) with 60.05 mIoU and 45.79 Tail-16 IoU. Among strong RGB-T baselines, AMDANet reports 68.13 mIoU and 54.77 Tail-16 IoU, MambaSeg reports 68.31 mIoU and 55.54 Tail-16 IoU, and CRM reports 67.78 mIoU and 55.46 Tail-16 IoU (Fan et al., 29 Apr 2026).

Relative to MambaSeg, GSCNet gains 2.73 mIoU and 4.63 Tail-16 IoU. Relative to RGB-only SegFormer, it gains 10.99 mIoU and 14.38 Tail-16 IoU. The reported interpretation is that naive or standard RGB-T fusion under misalignment can hurt or only modestly help, whereas explicit alignment and semantic graph calibration yield substantial gains, especially on tail classes (Fan et al., 29 Apr 2026).

The ablation structure isolates FDAM and SGCM. A baseline without FDAM or SGCM reports 67.63 mIoU and 52.67 Tail-16 IoU. Adding FDAM alone yields 70.13 mIoU and 57.23 Tail-16 IoU. Adding SGCM alone yields 70.30 mIoU and 58.75 Tail-16 IoU. The full model reaches 71.04 mIoU and 60.17 Tail-16 IoU. This indicates complementary contributions: FDAM primarily addresses spatial misalignment, whereas SGCM primarily strengthens semantic relational reasoning for rare and confusable categories (Fan et al., 29 Apr 2026).

Further ablations dissect each module. For FDAM, adding asymmetric decoupling encoders yields 68.84 mIoU, adding the decoupling losses yields 69.35 mIoU, and adding IAA on top yields 70.13 mIoU. For SGCM, using hierarchy only yields 68.88 mIoU and 55.93 Tail-16 IoU; co-occurrence only yields 69.08 mIoU and 56.35 Tail-16 IoU; hierarchy plus co-occurrence yields 69.81 mIoU and 57.76 Tail-16 IoU; adding residual adjacency yields 70.30 mIoU and 58.75 Tail-16 IoU. The paper’s reading of these ablations is that taxonomy and co-occurrence are complementary, and that learnable residual edges further benefit especially rare classes (Fan et al., 29 Apr 2026).

Qualitative analysis attributes double edges, ghosting, and displaced small targets to misaligned fusion without FDAM, and category confusions such as pole versus streetlight versus traffic light to missing semantic calibration without SGCM. Combining the two modules yields the best overall segmentation, though some thin occluding structures remain difficult (Fan et al., 29 Apr 2026).

GSCNet’s conceptual premise is that RGB and thermal modalities contain both shared geometry and modality-specific appearance. FDAM operationalizes this by restricting alignment to a shared structural subspace and by preserving modality-specific private features for recognition. The reported rationale is that direct alignment in the full feature space is unreliable because RGB and thermal differ strongly in appearance, so an offset estimator may otherwise confuse radiometric variation with geometric displacement (Fan et al., 29 Apr 2026).

SGCM, in turn, treats category labels as structured rather than flat. Taxonomy captures semantic proximity, co-occurrence captures contextual compatibility, and residual adjacency captures data-driven relations not already encoded by either prior. A plausible implication is that SGCM functions as a graph-regularized correction layer over the class space, using the current image’s pooled evidence to determine how category embeddings should influence one another before being projected back to pixels.

This use of graph reasoning intersects with several adjacent lines of work. In weakly supervised semantic segmentation, graph reasoning has been used to globally model inter-class dependencies and to incorporate textual knowledge into both classification and segmentation stages (Zhang et al., 2023). In node classification, graph calibration has been formalized as a confidence-calibration problem in which graph structure and nodewise similarity guide temperature-based correction, rather than segmentation-logit refinement (Tang et al., 2023). In graph-aware calibrated diffusion, calibration is framed at the level of controllable graph dynamics, with explicit dependence on spectral properties and graph-specific parameterization (Alpay et al., 19 Aug 2025). These works operate in different problem domains, but collectively suggest that “semantic calibration” can denote graph-mediated correction either of predictions, confidences, or dynamics, depending on the task.

The limitations explicitly stated for GSCNet concern efficiency, dependency on predefined priors, and modality scope. The model has 160.66M parameters, 208.07G FLOPs, and 16.62 FPS at batch size 1 and 640×512 input resolution, which may be restrictive for onboard real-time UAV deployment. The authors identify knowledge distillation and efficient backbones as future directions. SGCM also depends on a predefined hierarchical taxonomy and co-occurrence statistics derived from URTF, so transferring the method to different label spaces or more than two modalities would require new priors. FDAM is designed for realistic parallax under roughly overlapping 640×512 fields of view, and extension to cross-resolution RGB-T segmentation is identified as future work (Fan et al., 29 Apr 2026).

In aggregate, GSCNet can be understood as a graph-augmented multimodal segmentation architecture in which spatial misalignment correction and semantic relation modeling are treated as equally necessary components. Its reported empirical gains on fine-grained and tail categories indicate that, within the URTF setting, category-graph reasoning is most effective when paired with explicit feature-level alignment rather than used as a stand-alone semantic prior (Fan et al., 29 Apr 2026).