HyperKD: Cross-Spectral Distillation

• HyperKD is a cross-spectral knowledge distillation framework that transfers representations from a multispectral teacher to a hyperspectral student.
• It employs spectral range-based channel alignment, spatial-aware masking, and a specialized loss to optimize masked autoencoder pretraining.
• Empirical results demonstrate significant improvements in reconstruction metrics and downstream tasks, affirming its impact on hyperspectral imaging.

Searching arXiv for the HyperKD paper and closely related remote-sensing foundation-model work. HyperKD is a cross-spectral knowledge distillation framework for pretraining a hyperspectral masked autoencoder by transferring representations from a multispectral geospatial foundation model into a student tailored for EnMAP imagery. The method is formulated as an inverse domain-shift setting: the teacher, Prithvi-100M, is pretrained on multispectral HLS data with lower spectral dimensionality, while the student receives 218-band hyperspectral EnMAP inputs. HyperKD addresses this spectral mismatch through spectral range-based channel alignment, spatial feature-guided masking, and a specialized loss combining reconstruction and feature distillation (Matin et al., 13 Aug 2025).

1. Definition and problem setting

HyperKD departs from the conventional knowledge distillation pattern in which a larger or more expressive teacher transfers knowledge to a smaller student on essentially the same modality. In HyperKD, the teacher is spectrally simpler but pretrained on abundant multispectral observations, and the student is spectrally richer but trained on a much scarcer hyperspectral corpus. The paper frames this as an inverse form of knowledge transfer across different types of spectral data, or an inverse-shift spectral domain adaptation problem (Matin et al., 13 Aug 2025).

The motivating asymmetry is specific to Earth observation. Large pretrained geospatial models have been built on multispectral observations, whereas hyperspectral imagery remains comparatively data-limited and computationally expensive to model at foundation-model scale. HyperKD therefore asks whether high-level geospatial structure learned from abundant multispectral data can regularize a student operating on much higher spectral dimensionality. The paper’s answer is affirmative, provided that the teacher operates within its native spectral regime and the transfer is imposed at the feature level rather than through direct output matching (Matin et al., 13 Aug 2025).

This suggests that HyperKD is best understood not as a compression method, but as a modality-bridging pretraining scheme for representation transfer under severe spectral mismatch. A plausible implication is that its main scientific significance lies in reusing existing multispectral foundation models for hyperspectral representation learning rather than in reducing inference cost.

2. Architectural formulation

The overall pipeline is built on a masked autoencoder with a Vision Transformer backbone. The teacher is Prithvi-100M, described as a pretrained ViT-MAE geospatial foundation model trained on HLS multispectral imagery, and the student is a ViT-MAE adapted to EnMAP hyperspectral imagery (Matin et al., 13 Aug 2025). The teacher is used as a source of intermediate features, while the student is trained to reconstruct masked EnMAP inputs and align its internal representations with those of the teacher.

The student operates on full $224 \times 224 \times 218$ EnMAP tiles. The teacher receives a derived six-band representation synthesized from the same EnMAP scene so that it remains compatible with the teacher’s pretrained spectral regime. The teacher produces intermediate encoder features from a selected transformer block; the student produces its own hidden features at a corresponding block; and a fully connected layer maps teacher features to the student feature dimensionality for comparison (Matin et al., 13 Aug 2025).

The architectural parameters stated for the student-side MAE are specific. The encoder uses patch size $16 \times 16$, embedding dimension 768, 12 transformer layers, 12 attention heads, and MLP hidden dimension 3072. The decoder uses embedding dimension 512, 12 transformer layers, 16 attention heads, and MLP hidden dimension 2048. The paper refers to a baseline “BaseKD” with this configuration and states that the same backbone is used for the student and HyperKD as well (Matin et al., 13 Aug 2025).

The paper reports that the best teacher-student layer pairing was layer 8 of the 12-layer encoder and that, among the tested distillation losses, KL divergence performed best relative to L1 and Jensen–Shannon divergence (Matin et al., 13 Aug 2025). This places the distillation signal at an intermediate representational level rather than at the reconstruction output.

3. Spectral range-based channel alignment

A central difficulty is that the teacher is pretrained on HLS multispectral data using six spectral bands across three timestamps, whereas the student consumes 218-band EnMAP inputs at a single timestamp. These domains differ in channel count, wavelength support, and band definitions, so the teacher cannot directly process the student’s hyperspectral cube (Matin et al., 13 Aug 2025).

HyperKD resolves this by defining HLS and EnMAP band sets as

$B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$

and

$B_{\text{EnMAP}} = \{e_1, e_2, \ldots, e_{218}\}.$

For each HLS band $b_i$, whose spectral range is $[\lambda_i^{\min}, \lambda_i^{\max}]$, the method identifies the subset of EnMAP bands satisfying

$$E_i = \{ e_j \in B_{\text{EnMAP}} \mid \lambda_j^{\min} \geq \lambda_i^{\min} \ \text{and} \ \lambda_j^{\max} \leq \lambda_i^{\max} \}.$$

The aligned synthetic band is then formed by averaging the corresponding EnMAP bands: $$\hat{b}_i = \frac{1}{|E_i|} \sum_{e_j \in E_i} e_j.$$

The resulting synthetic six-band set is

$$\hat{B}_{\text{EnMAP}} = \{\hat{b}_1, \hat{b}_2, \ldots, \hat{b}_6\}.$$

The paper explicitly interprets this as a way of aggregating fine-grained hyperspectral bands into coarse multispectral bands that approximate the teacher’s expected spectral signatures (Matin et al., 13 Aug 2025). The full 218-band richness is retained in the student branch, while only the teacher input path is spectrally compressed.

Bandwise analysis reported in the paper shows that HyperKD improves reconstruction across all EnMAP bands relative to the student without KD, with larger gains in spectral regions overlapping HLS bands and smaller gains in non-overlapping regions (Matin et al., 13 Aug 2025). This suggests that the transfer signal is strongest where spectral correspondence is explicit.

4. Spatial-aware masking and saliency estimation

HyperKD modifies the masking stage of masked autoencoder pretraining. Standard MAEs use random masking; HyperKD argues that this is suboptimal for hyperspectral imagery because the most informative reconstruction targets are often spatially heterogeneous or structurally complex regions (Matin et al., 13 Aug 2025).

The method therefore computes patch significance scores using either Gabor filtering or wavelet transforms. The Gabor filter is written as

$$G(x, y; \lambda, \theta, \psi, \sigma, \gamma) = \exp\left(-\frac{x'^2 + \gamma^2 y'^2}{2\sigma^2}\right) \cos\left(2\pi \frac{x'}{\lambda} + \psi\right),$$

with

$16 \times 16$0

The wavelet transform is given as

$16 \times 16$1

For each patch $16 \times 16$2, the significance score is defined as either

$16 \times 16$3

If the input image has size $16 \times 16$4 and is patchified into $16 \times 16$5 patches, then the number of spatial patches is

$16 \times 16$6

Given masking ratio $16 \times 16$7, the method ranks patches in descending order of significance and masks the top $16 \times 16$8 patches with highest significance (Matin et al., 13 Aug 2025). In effect, the student is forced to reconstruct the most salient or difficult spatial content rather than arbitrary regions.

The paper reports a hybrid training schedule in the best setup: salient masking first, followed by random masking fine-tuning beginning at epoch 100, denoted as “Start Rand. Masking = epoch 100” (Matin et al., 13 Aug 2025). The best model uses Gabor-based masking on salient masked patches and later random masking.

This suggests that HyperKD treats masking not as a neutral pretext mechanism but as a structured curriculum over reconstruction difficulty. A plausible implication is that the masking policy is intended to amplify the benefit of feature transfer by placing distillation pressure on the most semantically informative regions.

5. Loss design and optimization objective

HyperKD uses a specialized reconstruction objective because the paper argues that standard pixel-wise losses such as MSE or MAE are insufficient for hyperspectral imagery when used alone (Matin et al., 13 Aug 2025). The reconstruction loss combines MSE and SSIM.

The MSE term is

$16 \times 16$9

The SSIM-based term is given as

$B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$0

The reconstruction loss is then

$B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$1

The total objective combines reconstruction and distillation: $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$2

The exact feature-level KL distillation equation is not explicitly typeset in the paper, but the text states that $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$3 is a feature distillation loss between projected teacher mid-layer features and student mid-layer features, and that KLD is used in the final model (Matin et al., 13 Aug 2025).

The paper does not report full values of $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$4, $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$5, $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$6, or $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$7, nor the exact optimizer hyperparameters such as learning rate or weight decay (Matin et al., 13 Aug 2025). This omission is material for exact reproducibility. It also leaves some implementation details partially specified, especially because some equations in the source are described as corrupted.

6. Data regime, training configuration, and empirical behavior

The study centers on EnMAP hyperspectral imagery. EnMAP originally has 224 bands spanning 420–2450 nm at 30 m spatial resolution; after removing 6 noisy bands, the study uses 218 bands (Matin et al., 13 Aug 2025). The main study region is California, and robustness experiments extend evaluation to Colorado and Kansas.

The California split includes 5000 training tiles, 500 validation tiles, 2000 tiles in Test Dataset 1, and another 2000 non-overlapping tiles in Test Dataset 2. A broader CA+CO+KS dataset uses 4625 training tiles and 1320 test tiles for Test Dataset 3 (Matin et al., 13 Aug 2025). Inputs are standardized to zero mean and unit variance per band, following Prithvi’s normalization scheme, and stored in Zarr format.

Training uses Adam with a custom learning-rate scheduler, an effective batch size of 32, one NVIDIA A100 80GB GPU, and a masking ratio of 75% during both training and testing (Matin et al., 13 Aug 2025). The paper also states that EnMAP cannot provide aligned three-timestamp stacks because of sparse revisit, so one EnMAP timestamp closest to one of the teacher HLS timestamps is selected (Matin et al., 13 Aug 2025).

Reconstruction results

The paper compares a plain student without KD, a baseline KD model using Huber reconstruction with L1 KD and random masking, and several HyperKD variants. The main reconstruction metrics reported are PSNR and SSIM (Matin et al., 13 Aug 2025).

Model/setup	Dataset	Reported result
Student	Test Dataset 1	PSNR 24.61, SSIM 0.55
BaseKD	Test Dataset 1	PSNR 27.10, SSIM 0.65
Best HyperKD	Test Dataset 1	PSNR 31.02, SSIM 0.77
Student	Test Dataset 2	PSNR 25.48, SSIM 0.52
BaseKD	Test Dataset 2	PSNR 27.56, SSIM 0.62
Best HyperKD	Test Dataset 2	PSNR 30.62, SSIM 0.73
Student	Test Dataset 3	PSNR 25.57, SSIM 0.41
BaseKD	Test Dataset 3	PSNR 31.73, SSIM 0.80
Best HyperKD	Test Dataset 3	PSNR 33.62, SSIM 0.85

The reported ablations further indicate that layer 8 was the best feature-matching depth, and that KLD outperformed L1 and JS divergence in the reduced setup: L1 gave PSNR 26.86 and SSIM 0.63, KLD gave 27.39 and 0.65, and JS gave 27.08 and 0.64 (Matin et al., 13 Aug 2025). The specialized reconstruction loss and guided masking contribute materially beyond KD alone.

Downstream tasks

The pretrained HyperKD encoder is frozen and paired with a lightweight CNN head for downstream evaluation (Matin et al., 13 Aug 2025). The tasks include land cover classification, crop type identification, and soil organic carbon prediction.

For California land cover classification using five NLCD classes, the mean Top-1 accuracy changes from 75.42% for Student+CNN to 75.49% for HyperKD+CNN, and mean mIoU from 55.46% to 55.62%. The paper notes larger classwise changes, including Cultivated Crops from 90.80% to 92.91% Top-1 and Other from 71.67% to 72.78%, with mIoU from 61.36% to 63.70% (Matin et al., 13 Aug 2025).

For the multi-region CA+CO+KS land-cover setup, gains are larger for some classes: Cultivated Crops is reported as 56.70% for Student, 77.18% for BaseKD, and 85.50% for HyperKD; Herbaceous is 35.43% for Student, 38.70% for BaseKD, and 46.25% for HyperKD (Matin et al., 13 Aug 2025).

For crop type identification using CDL classes in California, mean Top-1 accuracy improves from 67.41% to 73.27%, and mean mIoU from 41.66% to 48.60%. Shrubland improves from 40.31% to 55.93% Top-1 and from 34.74% to 46.00% mIoU; Evergreen Forest improves from 81.72% to 93.62% Top-1 and from 53.04% to 56.44% mIoU (Matin et al., 13 Aug 2025).

For soil organic carbon prediction over California, Colorado, and Kansas, HyperKD reports MAE 0.038 versus 0.045 for the student without KD and 0.046 for baseline KD (Matin et al., 13 Aug 2025).

7. Context, interpretation, and limitations

HyperKD is situated within the masked autoencoder and geospatial foundation-model literature rather than within standard compression-oriented KD alone. The paper explicitly builds on a Masked Autoencoder formulation and transfers knowledge from Prithvi, while also placing itself in the broader landscape of geospatial foundation models such as SatMAE and ScaleMAE (Matin et al., 13 Aug 2025). Related MAE-style and remote-sensing representation-learning work includes masked autoencoders (He et al., 2021), SatMAE (Cong et al., 2022), and Scale-MAE (Reed et al., 2022).

What distinguishes HyperKD from ordinary KD is not merely the use of feature matching, but the combination of three constraints that are simultaneously active: a teacher trained in a different spectral regime, a student with much higher spectral dimensionality, and self-supervised masked reconstruction rather than supervised logit matching (Matin et al., 13 Aug 2025). In that sense, HyperKD is closer to cross-domain representation transfer than to teacher-student compression.

Several limitations are explicit in the paper. Some equations are corrupted in the source, especially the SSIM expression and parts of the set notation. The exact KLD feature loss is not printed explicitly. Hyperparameter values for $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$8, $B_{\text{HLS}} = \{b_1, b_2, \ldots, b_6\}$9, $B_{\text{EnMAP}} = \{e_1, e_2, \ldots, e_{218}\}.$0, and $B_{\text{EnMAP}} = \{e_1, e_2, \ldots, e_{218}\}.$1, as well as some optimizer and masking/filter settings, are not provided. The study also assumes teacher and student imagery at the same spatial resolution, and the authors explicitly identify varying spatial resolutions as future work (Matin et al., 13 Aug 2025).

A further limitation is structural: the spectral alignment is based on averaging EnMAP bands within overlapping HLS wavelength intervals. Non-overlapping EnMAP bands therefore benefit only indirectly, and the paper itself reports weaker gains there (Matin et al., 13 Aug 2025). Temporal mismatch is also only partially addressed, because the original temporal structure of Prithvi is not fully exploited when only one EnMAP timestamp is available.

These caveats do not negate the paper’s main result, but they delimit its generality. The most defensible interpretation is that HyperKD demonstrates a workable path for reusing multispectral geospatial priors in hyperspectral MAE pretraining, with empirical gains in both reconstruction fidelity and several downstream tasks, while leaving open questions on temporal transfer, spatial-resolution mismatch, and fully specified optimization details (Matin et al., 13 Aug 2025).