Spectral Endmember Alignment (SERA)

• SERA is a module for hyperspectral reconstruction that uses a compact set of physically interpretable spectral endmembers as domain-invariant anchors.
• It leverages the ATGP algorithm to extract and normalize pure spectral signatures, ensuring robust alignment across source and target domains.
• A momentum-based update mechanism refines the endmember bank, integrating seamlessly with the Mean-Teacher framework to address domain shift and data scarcity.

Spectral Endmember Representation Alignment (SERA) is a physically grounded module for hyperspectral image (HSI) reconstruction that leverages a compact set of spectral endmembers to act as domain-invariant anchors. Integrated within the SpectralAdapt framework, SERA addresses domain shift and data scarcity by guiding predictions on both labeled and unlabeled data. Its design incorporates both interpretable spectral priors and momentum-driven updates, forming a distinctive mechanism for semi-supervised domain adaptation (SSDA) in imaging across heterogeneous sources (Wen et al., 17 Nov 2025).

1. Endmember Extraction from Labeled Data

SERA begins with the collection of all labeled hyperspectral pixels from both source and (limited) target images, aggregated into a single matrix $S \in \mathbb{R}^{n \times C}$, where $n = H \cdot W$ is the number of pixels and $C$ is the number of spectral bands. To represent the dominant "pure" spectra present in combined domains, SERA employs the Automated Target Generation Process (ATGP) [Plaza and Chang, 2006]:

• The first endmember $e_1$ is chosen as the spectrum with maximum $L_2$ norm: $e_1 = \arg\max_{s_n \in S} \|s_n\|_2$.
• For $k=2,\ldots,K$, endmember $e_k$ is determined by maximizing orthogonality to the span of previously selected endmembers:

$$P_{k-1} = E_{k-1}(E_{k-1}^\top E_{k-1})^{-1} E_{k-1}^\top$$

$e_k = \arg\max_{s_n\in S} \| (I - P_{k-1})s_n \|_2$

• After $K$ iterations, the $K$ endmembers are $$\text{ATGP}(S, K) = [e_1, \ldots, e_K]^\top \in \mathbb{R}^{K \times C}$$.
• Each endmember vector is then $L_2$-normalized: $E^0 = \text{Norm}(\text{ATGP}(S, K))$, so that $(E^0)_i = (E^0)_i/\|(E^0)_i\|_2$.

This extraction produces a physically interpretable spectral prototype set spanning observed variance in the labeled dataset (Wen et al., 17 Nov 2025).

2. Domain-Invariant Anchor Bank

The set $E \in \mathbb{R}^{K \times C}$ forms a fixed-size bank of spectral prototypes, with each row $e_k$ corresponding to a select endmember. By construction, these prototypes encapsulate axes of spectral variability present in both source and target domains. Unlike conventional learned parameters, these anchors are not updated by back-propagation. Instead, they are adaptively refined through an online momentum update, which maintains their interpretability and allows them to remain approximately domain-invariant as the model’s predictions evolve. This mechanism ensures robust cross-domain alignment in feature space, anchoring predictions to physically plausible spectra.

3. Momentum-Based Endmember Update

During each training iteration, SERA updates the endmember bank $E$ via a momentum rule. For every predicted hyperspectral cube $\hat{Y}$ (from both labeled and unlabeled batches), the process is as follows:

• Compute a sample-level descriptor $$z = \text{Norm}(\text{AvgPool}(\hat{Y})) \in \mathbb{R}^C$$, where $\text{AvgPool}$ averages over spatial dimensions and $\text{Norm}$ denotes $L_2$-normalization.
• Assign $z_i$ to the nearest anchor via maximum cosine similarity: $a_i = \arg\max_{j=1...K} z_i^\top e^{t-1}_j$.
• Aggregate all $z_i$ assigned to anchor $k$ as $\mathcal{B}^k_t = \{ i | a_i = k \}$ and compute the batch mean $$\bar{z}^k_t = (1/|\mathcal{B}^k_t|) \sum_{i\in\mathcal{B}^k_t} z_i$$.
• Update each endmember by exponential moving average:

$$e^k_t \leftarrow \text{Norm}( m \cdot e^k_{t-1} + (1-m)\cdot \bar{z}^k_t )$$

where $m$ is the momentum coefficient (typically $m_\text{end} = 0.9$).

The momentum update ensures that endmembers track the slow evolution of the prediction space, while smoothing out noisy batch statistics.

4. Anchor-Guided Prediction and SERA Loss

SERA imposes domain structure on model predictions by encouraging them to align closely with the learned endmembers:

• For each predicted descriptor $z_i$ from the student network, the goal is proximity to at least one anchor, quantified by the SERA loss:

$$\mathcal{L}_\text{SERA} = \frac{1}{|\mathcal{B}|} \sum_{i\in\mathcal{B}} \left[ 1 - \max_{k=1...K}(z_i^\top e_k^{t-1}) \right ]$$

• Here, both $z_i$ and $e_k$ are unit vectors, so $z_i^\top e_k = \cos \theta \in [-1, 1]$; minimizing $1 - \max_k \cos \theta$ encourages sample descriptors to cluster tightly around one or more endmembers.

In SpectralAdapt, $\mathcal{L}_\text{SERA}$ is applied to all student predictions on unlabeled data, acting as an unsupervised spectral alignment regularizer (Wen et al., 17 Nov 2025).

5. Joint Loss Formulation and Integration with SpectralAdapt

SERA is integrated into SpectralAdapt’s Mean-Teacher framework as follows:

• The training objective is the sum of supervised and unsupervised losses:

$$\mathcal{L}_\text{sup} = \text{L}_1(\hat{Y}_\text{student}, Y) + \text{SSIM}(\hat{Y}_\text{student}, Y)$$

$$\mathcal{L}_\text{con} = \text{L}_1(\hat{Y}_\text{student}, \hat{Y}_\text{teacher})$$

$$\mathcal{L}_\text{un} = \lambda_\text{un} \cdot \mathcal{L}_\text{con} + (1-\lambda_\text{un}) \cdot \mathcal{L}_\text{SERA}$$

$$\mathcal{L}_\text{total} = \mathcal{L}_\text{sup} + \mathcal{L}_\text{un}$$

with typical settings: $\lambda_\text{un}=0.3$, Mean-Teacher momentum $m_\text{ema}=0.99$, and endmember momentum $m_\text{end}=0.9$.

• Training alternates between updating the student parameters by $\nabla_\theta \mathcal{L}_\text{total}$, updating the teacher by momentum, and updating the endmember bank by its specific momentum rule.

The following table summarizes the central loss components and their weights in SpectralAdapt:

Loss Term	Description	Weight
$\mathcal{L}_\text{sup}$	Supervised L1 + SSIM reconstruction	$\lambda_\text{sup}$
$\mathcal{L}_\text{con}$	Consistency (student vs teacher) on unlabeled	$\lambda_\text{un}$
$\mathcal{L}_\text{SERA}$	Cosine alignment to endmember anchors	$1-\lambda_\text{un}$

6. Algorithmic Workflow and Pseudocode

The SERA workflow is executed within each iteration of SpectralAdapt training:

1. Initialization: Construct the initial endmember bank $E^0$ using ATGP on all labeled pixels and normalize.
2. Batch Processing: For each mini-batch:
   • Forward pass through student and teacher networks, applying spectral density masking and augmentations.
   • Compute supervised and consistency losses.
   • Extract spectral descriptors from student predictions and compute SERA loss.
   • Combine terms into total loss and update student network.
   • Update teacher model via EMA (Exponential Moving Average).
   • Assign descriptors to nearest endmember anchors and update the endmember bank using the momentum rule.

The complete process is detailed in the stepwise pseudocode provided in (Wen et al., 17 Nov 2025). Key algorithmic steps ensure that SERA anchors, rather than being static, adapt smoothly as the spectral distribution of predictions evolves while remaining physically interpretable.

7. Context and Significance

SERA’s design addresses the core challenges of domain shift and limited labeled data in hyperspectral reconstruction, which are prevalent in healthcare scenarios where acquiring exhaustive HSI datasets is impractical. By deriving domain-invariant anchors grounded in physical spectral endmembers and imposing alignment via a rigorous loss, SERA facilitates generalization across differing patient populations and imaging conditions. The module’s explicit momentum-based update differs from ubiquitous back-propagation, decoupling physical anchor adjustment from the main gradient flow. This suggests a new avenue for integrating spectral domain priors into semi-supervised learning pipelines (Wen et al., 17 Nov 2025).

A plausible implication is that physically motivated anchor banks such as those used in SERA can provide greater interpretability and domain robustness compared to standard learned feature prototypes, especially where available labeled spectra are scarce or span multiple acquisition conditions.

For detailed implementation guidelines, reference equations, and experimental validation, see "SpectralAdapt: Semi-Supervised Domain Adaptation with Spectral Priors for Human-Centered Hyperspectral Image Reconstruction" (Wen et al., 17 Nov 2025).