SpectralAdapt (SSDA) for HSI Reconstruction

• SpectralAdapt (SSDA) is a semi-supervised domain adaptation framework that reconstructs hyperspectral images using spectral priors like SDM and SERA.
• It employs a Mean-Teacher architecture with both weakly and strongly augmented views to enforce consistency and enhance training stability.
• Experimental results show up to 4.9% SSIM improvement and +5.6 dB PSNR gain, proving its effectiveness in label-scarce and high domain shift scenarios.

SpectralAdapt (SSDA) is a semi-supervised domain adaptation framework for hyperspectral image (HSI) reconstruction, specifically designed to address the challenges inherent in medical and human-centered hyperspectral datasets characterized by data scarcity, strong domain shift, and limited labeled target data. The method integrates spectral priors via two primary modules—Spectral Density Masking (SDM) and Spectral Endmember Representation Alignment (SERA)—and leverages the Mean-Teacher paradigm. The approach demonstrably improves spectral fidelity, cross-domain generalization, and training stability in human-centered HSI reconstruction tasks using accessible modalities such as RGB inputs (Wen et al., 17 Nov 2025).

1. Core Architecture and Workflow

The SpectralAdapt (SSDA) framework employs a Mean-Teacher architecture with two identical RGB-to-HSI networks (MST++ backbone):

• Student network ($f_\theta$): Parameters $\theta$ updated via gradient descent.
• Teacher network ($f_{\theta'}$): Parameters $\theta'$ updated as the exponential moving average (EMA) of $\theta$, with $m_{\text{ema}} = 0.99$:

$$\theta' \leftarrow m_{\rm ema}\,\theta' + (1 - m_{\rm ema})\,\theta$$

• Data flow:
  • Labeled source and target samples: Both networks receive weak augmentation; supervised reconstruction loss is computed.
  • Unlabeled target samples: The student receives weak augmentation and SDM masking; the teacher receives strong augmentation. Consistency is enforced between student and teacher predictions on the same input via an $L_1$ loss.
  • All network outputs are globally pooled and aligned to a dynamic endmember bank via SERA.
• No adversarial losses: The method does not use separate discriminators or GANs; domain adaptation is enforced purely by consistency training and spectral prior alignment.

2. Spectral Density Masking (SDM)

SDM is a spectral reasoning module that enhances model robustness and cross-domain consistency by leveraging the spectral complexity of RGB channels:

• Spectral-complexity calculation: For each RGB channel $b \in \{\mathrm{R}, \mathrm{G}, \mathrm{B}\}$, construct a perturbed HSI cube $\mathbf{S}^{(b)}$ where $I_b$ (the indices for band $b$) are set to the channel-average $\bar S_c$, all others remain. The spectral density of masking channel $b$ is

$$\mathcal{D}_b = \frac{1}{N}\sum_{n=1}^N \arccos \left( \frac{ \langle \mathbf{S}_n^{(b)}, \mathbf{S}_n \rangle }{ \| \mathbf{S}_n^{(b)} \|_2 \| \mathbf{S}_n \|_2 + \varepsilon } \right )$$

• Adaptive masking: Compute per-channel masking ratios

$$r_b = r_{\min} + \frac{\mathcal{D}_b - \min(\mathcal{D})}{\max(\mathcal{D}) - \min(\mathcal{D})} (r_{\max} - r_{\min})$$

where $r_{\min} = 0.1$, $r_{\max} = 0.9$, and optimal average mask rate is around 70%.

• Block random masking: Each channel is divided into $s \times s$ blocks; a fraction $r_b$ of the blocks in each channel are masked.
• Consistency integration: The student model receives the masked, weakly augmented view; the teacher receives a strongly augmented view; the consistency loss is:

$$\mathcal{L}_{\rm con} = \| \hat y_j^T - \hat y_j^{T'} \|_1$$

3. Spectral Endmember Representation Alignment (SERA)

SERA enforces global feature alignment to physically meaningful domain-invariant spectral anchors:

• Endmember extraction: From all labeled pixels, $K$ spectral endmembers $\{\mathbf{e}_k\}$ are extracted using Automated Target Generation Process (ATGP):

$$\mathbf{e}_1 = \arg\max_n \| \mathbf{s}_n \|_2, \qquad \mathbf{e}_k = \arg\max_n \| (\mathbf{I} - \mathbf{P}_{k-1}) \mathbf{s}_n \|_2$$

where $\mathbf{P}_{k-1}$ projects onto previously selected endmembers.

• Feature-anchor alignment: Each output HSI is global-pooled and normalized to $\mathbf{z}$, and assigned to the nearest endmember anchor. The SERA loss is

$$\mathcal{L}_{\rm SERA} = \frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} (1 - \max_k \mathbf{z}_i^\top \mathbf{e}_k)$$

• Momentum anchor update: Endmember anchors are updated each iteration by momentum averaging with current assigned batch features ($m_{\text{end}} = 0.9$).

4. Optimization Objective and Training Dynamics

The total loss comprises supervised and unsupervised components:

• Supervised loss: For labeled data (source and target),

$$\mathcal{L}_{\rm sup}^{\rm total} = \sum_{(x,y)\in \mathcal{D}_l^S \cup \mathcal{D}_l^T} \left[ \lambda_{\rm sup} \mathcal{L}_1(\hat y, y) + (1-\lambda_{\rm sup}) \mathcal{L}_{\rm SSIM}(\hat y, y) \right]$$

with $\lambda_{\rm sup} = 0.4$.

• Unsupervised loss: For unlabeled target data,

$$\mathcal{L}_{\rm un}^{\rm total} = \lambda_{\rm un} \mathcal{L}_{\rm con} + (1-\lambda_{\rm un}) \mathcal{L}_{\rm SERA}$$

with $\lambda_{\rm un} = 0.3$.

• Overall objective:

$$\mathcal{L}_{\rm total} = \mathcal{L}_{\rm sup}^{\rm total} + \mathcal{L}_{\rm un}^{\rm total}$$

Training involves mini-batching, gradient-based updates of the student, EMA updates of the teacher, and per-iteration anchor bank momentum updates.

5. Experimental Validation and Performance

Benchmark experiments on cross-domain HSI reconstruction (e.g., NTIRE $\to$ Hyper-Skin) with 1.5% labeled target data demonstrate:

Method	SSIM (%)	SAM (deg)	PSNR (dB)
Mean-Teacher	85.3	23.05	23.23
+SDM	87.86	–	–
+SERA	89.36	–	–
SSDA (SDM+SERA)	90.24	17.11	28.78

• The joint use of SDM and SERA yields 4.9% SSIM improvement, 5.9° SAM reduction, and +5.6 dB PSNR gain over the baseline Mean-Teacher model.
• SDM-only and SERA-only ablations both yield substantial improvements; optimal SDM mask rate is around 70%.
• Downstream medical segmentation (Choledoch, HeiPorSPECTRAL): SSDA's reconstructed HSI achieves mIoU competitive with raw HSI despite label scarcity (Choledoch: 79.3% mIoU vs. 81.7% for raw HSI), outperforming direct RGB-based segmentation.

6. Significance, Context, and Directions

SpectralAdapt establishes a spectral prior-guided semi-supervised domain adaptation paradigm for HSI reconstruction in data-scarce, high-variance medical settings:

• Domain adaptation via spectral structure: The method avoids explicit adversarial losses, instead relying on spectral density masking to focus consistency regularization on complex spectral regions, and SERA to impose a physically interpretable global feature structure across domains.
• Practicality in label-limited regimes: SSDA leverages abundant unlabeled and scarce labeled target data, showing substantial performance gains even with 1.5% labeled target pixels.
• Transferability and robustness: The integration of dynamic endmember anchors and adaptive masking rates permits robust adaptation across large domain shifts, indicating application potential for diverse medical HSI tasks.
• Comparison to alternatives: Unlike self-training or explicit adversarial DA frameworks, SpectralAdapt exploits domain-invariant spectral physics and demonstrates enhanced spectral fidelity and generalization by embedding prior knowledge into the training and adaptation processes (Wen et al., 17 Nov 2025).

The approach positions spectral prior-guided semi-supervised domain adaptation as an efficient and scalable solution for healthcare HSI imaging, with the spectral prior modules (SDM and SERA) enabling effective mitigation of both domain shift and spectral reconstruction degradation.