Representative Spectral Correlation Network (RSCNet)

• RSCNet is a multi-source remote sensing classifier that dynamically selects a representative subset of hyperspectral bands to reduce spectral redundancy.
• It integrates HSI with SAR/LiDAR modalities via a cross-source adaptive fusion module that balances material discrimination and structural cues.
• Empirical evaluations show that RSCNet enhances land-cover classification accuracy while reducing computational complexity across diverse datasets.

Representative Spectral Correlation Network (RSCNet) is a multi-source remote sensing image classification framework introduced for land-cover classification from hyperspectral image (HSI) data jointly with SAR or LiDAR, with the explicit aim of addressing two coupled difficulties: spectral redundancy in high-dimensional HSI and heterogeneous characteristics across sensing modalities (Gong et al., 30 Apr 2026). In this formulation, the “representative spectral correlation” idea does not denote a graph-Laplacian spectral method; it denotes task-adaptive selection of representative original HSI bands and their interaction with cross-source structural cues. The model centers on two components: a Key Band Selection Module (KBSM), which selects informative HSI bands under cross-source guidance, and a Cross-source Adaptive Fusion Module (CAFM), which performs adaptive weighting and local-global contextual refinement for feature fusion (Gong et al., 30 Apr 2026).

1. Problem setting and conceptual scope

RSCNet is designed for supervised multi-source remote sensing classification on three settings: HSI+SAR on Augsburg and Berlin, and HSI+LiDAR on Houston2013 (Gong et al., 30 Apr 2026). The underlying rationale is that the modalities are complementary. HSI provides rich and continuous spectral signatures and is suited to fine material discrimination, whereas SAR and LiDAR provide structural, geometric, elevation, backscatter, or polarization cues and are often more robust to illumination or weather effects. In the paper’s task framing, HSI indicates what material is present, while SAR/LiDAR indicates how it is structured (Gong et al., 30 Apr 2026).

The method is motivated by two specific limitations of prior fusion pipelines. First, HSI contains hundreds of contiguous bands, which produces strong inter-band correlation, noisy or redundant channels, increased computational cost, and a greater risk of overfitting. Second, HSI and SAR/LiDAR arise from different sensing mechanisms and therefore occupy different semantic and statistical spaces, so simple concatenation, summation, or generic attention may not align them effectively (Gong et al., 30 Apr 2026).

Within this framework, “representative spectral correlation” means that the network should not treat all HSI bands uniformly. Instead, it should identify a subset of representative spectral bands whose interactions are less redundant, more discriminative, and especially relevant when interpreted jointly with the auxiliary modality. This suggests that the model’s central “spectral” operation is band selection and cross-source interaction, not eigendecomposition of an affinity graph (Gong et al., 30 Apr 2026).

2. End-to-end architecture

RSCNet takes three inputs: original HSI, PCA-reduced HSI, and SAR/LiDAR (Gong et al., 30 Apr 2026). The architecture first applies PCA to the original HSI, then encodes the three streams into

$$\mathbf{F}_h \in \mathbb{R}^{hw \times c}, \qquad \mathbf{F}_r \in \mathbb{R}^{hw \times r}, \qquad \mathbf{F}_x \in \mathbb{R}^{hw \times r},$$

where $c$ is the number of original HSI bands, $r$ is the reduced spectral dimension, and $hw$ is the flattened spatial feature dimension (Gong et al., 30 Apr 2026).

The PCA-reduced HSI feature $\mathbf{F}_r$ and auxiliary feature $\mathbf{F}_x$ are fused by CAFM to obtain an initial fused representation $\mathbf{F}_{fus}$, which is then passed through an FFN for nonlinear transformation (Gong et al., 30 Apr 2026). KBSM then uses $\mathbf{F}_{fus}$ to guide selection of important spectral tokens from the original HSI feature $\mathbf{F}_h$, producing $\mathbf{F}'_h$. The tuple $c$0 is then fed into the Representative Spectral Correlation Block (RSCB), repeated $c$1 times for iterative cross-source refinement. The final stage applies a cross-attention fusion and a 2-layer MLP classifier to generate class predictions (Gong et al., 30 Apr 2026).

Several implementation choices are explicitly analyzed. The best patch size is reported as $c$2, balancing useful context against background redundancy, and the preferred number of RSCB blocks is $c$3, since performance improves or remains stable up to around four blocks and then plateaus or slightly decreases (Gong et al., 30 Apr 2026). The paper also reports that removing PCA increases parameter count and reduces accuracy, so PCA remains an efficiency-preserving preprocessing stage rather than being discarded entirely (Gong et al., 30 Apr 2026).

3. Key Band Selection Module

KBSM is the mechanism that operationalizes the “representative spectral” part of RSCNet. Its purpose is to reduce HSI redundancy without relying solely on a static, task-agnostic transformation such as PCA. The module receives hyperspectral features $c$4 and auxiliary fused features $c$5, computes a cross-source-guided attention map, applies dynamic sparse gating, and finally performs discrete top-$c$6 band selection (Gong et al., 30 Apr 2026).

The paper gives the KBSM procedure explicitly: $c$7

$c$8

$c$9

$r$0

$r$1

$r$2

Here, $r$3 is a band-importance vector, $r$4 is the Dynamic Sparse Gating Module output, and $r$5 is the selected-band feature (Gong et al., 30 Apr 2026).

A central property of KBSM is that its final selection is discrete rather than merely soft. The scores are continuous up to $r$6, but the use of $r$7 followed by $r$8 means that the final representation consists of hard-selected original bands. This distinguishes KBSM from PCA, which linearly projects all bands into a reduced basis and may remove low-variance but class-discriminative spectral cues (Gong et al., 30 Apr 2026).

The selected dimensionality is controlled by key band ratio $r$9. The paper tests $hw$0 from 10% to 70% and reports best settings of 50% on Augsburg, 20% on Berlin, and 20% on Houston2013 (Gong et al., 30 Apr 2026). The selected-band features are evaluated through average Pearson correlation between bands (ACC, lower is better for redundancy reduction) and mutual information with labels (MI, higher is better for discriminability). After KBSM, ACC decreases and MI increases on all three datasets. For Augsburg, ACC changes from 0.4342 to 0.3732 and MI from 0.4192 to 0.4495; for Berlin, ACC changes from 0.6537 to 0.6172 and MI from 0.3241 to 0.3569; for Houston2013, ACC changes from 0.7452 to 0.7179 and MI from 0.7914 to 0.8308 (Gong et al., 30 Apr 2026). These measurements support the paper’s claim that KBSM suppresses redundancy while preserving more discriminative spectral structure.

4. Cross-source Adaptive Fusion Module

CAFM addresses the heterogeneous fusion problem by combining PCA-reduced HSI and SAR/LiDAR in two stages: cross-source attention weighting and local-global contextual refinement (Gong et al., 30 Apr 2026). The module is described procedurally rather than through a full printed symbolic equation set in the available methodology text.

In the first stage, HSI is projected with a 3D convolution and SAR/LiDAR with a 2D convolution, after which the transformed features are concatenated. Global average pooling and two convolution layers generate source attention descriptors, and a Softmax normalizes them into source weights. These weights are then applied to the modality-specific features before summation, yielding an intermediate fused representation (Gong et al., 30 Apr 2026). This design is intended to let the network decide how much each modality should contribute in different contexts.

In the second stage, the intermediate feature is refined through parallel local and global attention branches. The local branch emphasizes spatial textures, object boundaries, and fine-grained patterns, while the global branch captures long-range dependencies and context aggregation. Their outputs are combined, normalized, and used to reweight the intermediate feature, after which the refined output is added back to form the final fused representation $hw$1 (Gong et al., 30 Apr 2026). A plausible implication is that CAFM is designed to align material cues from HSI with structural cues from SAR/LiDAR at both short and long spatial scales.

The ablation results show that CAFM improves performance consistently even when used alone. Relative to the baseline without KBSM or CAFM, adding CAFM changes the reported scores from 90.19 to 91.13 on Augsburg, from 72.27 to 73.35 on Berlin, and from 89.45 to 91.31 on Houston2013 (Gong et al., 30 Apr 2026). The gains are smaller than those from KBSM on some datasets, but the full model performs best when both modules are used together.

5. Empirical evaluation and computational profile

RSCNet is evaluated against ten multisource fusion baselines: FusAtNet, $hw$2ENet, AsyFFNet, ExViT, HCT, MACN, MICF-Net, GCCQTNet, CHNet, and MGMNet (Gong et al., 30 Apr 2026). The datasets are heterogeneous in both modality and difficulty. Augsburg uses HySpex HSI with 180 spectral bands and Sentinel-1 SAR with four polarization-derived features on a $hw$3 scene with 7 classes. Berlin uses simulated HyMap/EnMAP-like HSI with 244 bands and Sentinel-1 dual-polarization VV/VH SLC, with nearest-neighbor interpolation for alignment and 8 classes. Houston2013 uses 144-band HSI together with LiDAR over a $hw$4 scene with 15 classes (Gong et al., 30 Apr 2026).

The main results are reported in terms of OA, AA, and Kappa. On Augsburg, RSCNet attains OA 91.50%, AA 67.89%, and Kappa 0.8776, improving over the second-best OA of 91.18% by 0.32% (Gong et al., 30 Apr 2026). On Berlin, it reaches OA 78.19%, AA 61.98%, and Kappa 0.6544, improving over the second-best OA of 77.13% by 1.06% (Gong et al., 30 Apr 2026). On Houston2013, it achieves OA 92.66%, AA 93.76%, and Kappa 0.9204, improving over the second-best OA of 90.98% by 1.68% and over the second-best Kappa of 0.9020 by 0.0184 (Gong et al., 30 Apr 2026).

The module-level ablations indicate complementarity. The baseline scores are 90.19, 72.27, and 89.45 on Augsburg, Berlin, and Houston2013; adding CAFM alone yields 91.13, 73.35, and 91.31; adding KBSM alone yields 91.37, 76.74, and 91.69; and combining KBSM with CAFM yields 91.50, 78.19, and 92.66 (Gong et al., 30 Apr 2026). Additional ablations show that KBSM outperforms other band-selection methods on Berlin, with reported OA values of 74.50 for ISSC, 75.74 for OPBS, 76.51 for BAM, 76.08 for SRL-SOA, 77.43 for LGCAF, and 78.19 for KBSM (Gong et al., 30 Apr 2026).

The computational profile reported on Augsburg is 2.8812 M parameters, 0.3346 G FLOPs, and 0.3483 s inference time (Gong et al., 30 Apr 2026). The paper emphasizes that this is substantially smaller than several heavier baselines such as FusAtNet at 36.8236 M parameters, CHNet at 20.7745 M, and MGMNet at 6.1251 M, while still achieving the best accuracy (Gong et al., 30 Apr 2026). Qualitative analyses, including classification maps and t-SNE visualizations, are reported to show cleaner and more coherent maps and clearer class separation after KBSM-based selection (Gong et al., 30 Apr 2026).

6. Interpretation, limitations, and related terminology

RSCNet’s main contribution is the coupling of cross-source-guided spectral selection with heterogeneous fusion. KBSM retains actual original HSI bands instead of only projected PCA components, while CAFM learns how structural auxiliary data should influence representation formation. The resulting interpretation is that the model reduces redundancy not by generic compression alone, but by selecting a representative subset of original spectral tokens whose usefulness is evaluated relative to the fused multisource context (Gong et al., 30 Apr 2026).

The reported limitations are equally important. The method still uses PCA preprocessing before CAFM, so it is not a fully PCA-free pipeline. The paper also notes future work on robustness under atmospheric noise and cross-sensor variations, which suggests that these factors remain open issues. In addition, exact behavior under severe misregistration is not deeply analyzed, and some class-wise accuracies remain unstable on highly imbalanced datasets, especially in Berlin (Gong et al., 30 Apr 2026).

A common misconception is to interpret RSCNet as a graph-spectral network in the sense of Laplacian eigendecomposition or adjacency spectral embedding. That is not the meaning of “spectral” in this model. Here, “spectral” refers primarily to HSI bands and their representative selection. This differs from spectral-analysis-based clustering architectures such as SANet, which stack spectral embedding modules over affinity graphs for unsupervised image clustering (Wang et al., 2020). It also differs from the unrelated WiFi sensing model named RSCNet, where the acronym expands to Real-time Sensing and Compression Network rather than Representative Spectral Correlation Network (Barahimi et al., 2024).

In that sense, RSCNet is best understood as a supervised multi-source remote sensing classifier whose distinctive idea is to preserve representative original spectral structure while adaptively aligning it with heterogeneous structural modalities. Its empirical profile indicates that this design is effective across both HSI+SAR and HSI+LiDAR settings and does so with moderate model complexity (Gong et al., 30 Apr 2026).