TerraMAE: Self-Supervised HSI Learning

• TerraMAE is a self-supervised framework for hyperspectral satellite imagery that learns spatial-spectral embeddings via adaptive masked autoencoding.
• It employs an enhanced composite loss combining MAE, SSIM, and SID to accurately preserve pixel, spatial, and spectral information.
• The framework achieves state-of-the-art performance on crop identification, land cover classification, and soil texture prediction with robust transferability.

Searching arXiv for TerraMAE and closely related MAE background papers. TerraMAE is a self-supervised learning framework for hyperspectral satellite imagery (HSI) that learns spatial-spectral embeddings through adaptive masked autoencoding. It is designed for Earth observation settings in which imagery provides sub-30 m views of the surface across hundreds of contiguous spectral bands, enabling fine-grained analysis of soils, crops, and land cover. The framework targets a central limitation of standard masked autoencoders in hyperspectral regimes: they do not adequately exploit the intricate correlations that jointly span image space and spectral dimension in 200+ band data. TerraMAE addresses this through adaptive channel grouping based on statistical reflectance properties and an enhanced reconstruction loss that combines pixel-wise, spatial, and spectral criteria, and it is evaluated on crop identification, land cover classification, and soil texture prediction (Faruk et al., 9 Aug 2025).

1. Problem setting and design rationale

TerraMAE is motivated by four properties of hyperspectral Earth observation data. First, HSIs exhibit strong spatial-spectral correlation: land cover transitions, field boundaries, and soil variability are expressed not only in spatial texture but also in subtle reflectance signatures across contiguous bands. Second, high spectral dimensionality makes naive masking and grouping strategies, which are often acceptable for RGB or low-band multispectral imagery, suboptimal for HSI. Third, labeled data are scarce relative to the geographic scope of remote sensing archives, which constrains purely supervised deep learning. Fourth, many operational geospatial tasks require generalizable representations rather than task-specific encoders (Faruk et al., 9 Aug 2025).

Within this setting, TerraMAE is formulated as a masked autoencoder specialized for HSI rather than as a direct adaptation of RGB-oriented MAE pipelines. The framework is explicitly intended to produce representations that transfer to varied downstream problems, including crop type, land cover, and soil texture, while using lightweight decoders and minimal tuning. This suggests that the model is positioned not merely as a reconstruction system but as a pretraining mechanism for broad geospatial transfer.

2. Architectural formulation

TerraMAE retains the masked autoencoding paradigm but modifies it in two principal ways: adaptive channel grouping with independent masking, and an enhanced composite reconstruction loss. The high-level workflow consists of patch-wise masking in which 75% of input data patches are masked, with masking performed independently within spectral groups; channels are clustered into groups and processed independently; dual positional embeddings combine 2D spatial and 1D spectral group position information; a Vision Transformer-Large backbone serves as encoder; and a reconstruction decoder reconstructs masked content (Faruk et al., 9 Aug 2025).

The encoder is specified as a ViT-L backbone with a 1024-dimensional embedding, 24 blocks, and 16 heads. This backbone choice situates TerraMAE within the transformer-based HSI representation learning family, but the distinctive element is not the encoder alone. The operative claim is that masking and representation learning must respect the structure of the spectral axis rather than treating channels as a homogeneous stack.

The independent masking over adaptive channel groups is especially consequential. Blind masking across all channels can remove critical spatial and spectral structure simultaneously, whereas grouped masking permits the model to preserve and reconstruct correlated portions of the spectrum in a more controlled way. A plausible implication is that this setup encourages the latent representation to encode inter-group dependencies instead of merely interpolating within local wavelength neighborhoods.

3. Adaptive channel grouping and the Spectral Comparison Index

A defining component of TerraMAE is its adaptive channel grouping strategy, which is based on the Spectral Comparison Index (SCI). The stated motivation is that fixed wavelength splits and other static grouping heuristics do not generalize well to sensor-specific or dataset-specific spectral structure in HSI. TerraMAE therefore introduces a data-driven similarity measure between spectral bands derived from statistical reflectance properties (Faruk et al., 9 Aug 2025).

For bands $i$ and $j$, with reflectance values $I_i$ and $I_j$, the SCI at each spatial location is defined as

$$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$

The index ranges over $[0,1]$, with larger values indicating greater similarity. For each channel, TerraMAE computes a mean reflectance image over the training data, forms SCI maps comparing pairs of mean images across space, and aggregates them into a single channel-pair score:

$SCI_{prod} = SCI_{\mu} \times (1 - SCI_{\sigma}).$

Here, $SCI_{\mu}$ is the mean SCI across the dataset and $SCI_{\sigma}$ is the standard deviation, used to downweight unstable similarity. The intended effect is to group bands that are not only similar but stably similar across the dataset.

Channels are then clustered by maximizing $SCI_{prod}$, yielding adaptive groups that are not necessarily adjacent in wavelength and are not necessarily uniform in size. The framework compares this strategy to several alternatives: k-Means and HAC based on channel-level statistical descriptors, and domain-informed groupings such as VNIR-SWIR and Soil Reflectance splits. According to the reported results, SCI grouping consistently yields the best performance both in downstream utility and in statistical silhouette scores relative to these alternatives (Faruk et al., 9 Aug 2025).

This grouping mechanism is significant because it encodes a view of HSI structure that is simultaneously spectral and spatial. The paper states that SCI grouping allows the model to exploit spatially consistent spectral similarity and to align architecture with real-world geospatial patterns. This suggests a departure from purely wavelength-centric grouping toward correlation structures that emerge empirically from the training corpus.

4. Composite reconstruction objective

TerraMAE supplements standard pixel reconstruction losses with explicit spatial and spectral quality terms. The paper argues that conventional MAE or MSE objectives do not adequately enforce either spatial structure or spectral signature preservation, both of which are central to HSI fidelity (Faruk et al., 9 Aug 2025).

The composite loss is

$j$0

with tuned default weights $j$1.

The spatial term uses the Structural Similarity Index:

$j$2

together with the normalized loss form

$j$3

The spectral term uses Spectral Information Divergence:

$j$4

where $j$5, and the normalized form is

$j$6

with $j$7.

The paper describes the three components as enforcing pixel-wise accuracy, spatial edge and texture retention, and spectral signature preservation, respectively. Training further uses linear scheduling for the loss weights, beginning with a stronger pixel focus and gradually increasing SSIM and SID contributions for stability and convergence. In the reported ablation, the best loss configuration, $j$8, $j$9, and $I_i$0, attains reconstruction MAE $I_i$1, PSNR $I_i$2, and SSIM $I_i$3, compared with MAE-only training at MAE $I_i$4 and SSIM $I_i$5 (Faruk et al., 9 Aug 2025).

5. Reconstruction fidelity and information preservation

TerraMAE is evaluated as a reconstruction model with emphasis on preserving both spatial and spectral information. The reported outcomes are lower error and greater structural fidelity than both MAE and ResNet-50 baselines, including sharper field boundaries and subtler land-cover transitions (Faruk et al., 9 Aug 2025).

On held-out geographies, TerraMAE is reported to achieve SSIM $I_i$6, and in reconstruction tests it attains MAE $I_i$7 and SSIM $I_i$8, compared with a baseline MAE of $I_i$9 and SSIM of $I_j$0. The paper also states that mean absolute error is reduced by up to 73% versus a no-grouping MAE and that the model maintains $I_j$1 SSIM and $I_j$2 MAE on both seen data from California and unseen test data such as Colorado and Kansas (Faruk et al., 9 Aug 2025).

These results are used to support the claim that TerraMAE preserves information in ways that are not captured by raw pixel error alone. The composite loss is presented as producing reconstructions that are more faithful in perceptual and physically meaningful terms, while the SCI grouping is presented as enabling the encoder to exploit spatially coherent spectral similarity.

6. Downstream tasks, transfer behavior, and trade-offs

The practical utility of the learned representations is assessed on three downstream geospatial tasks using a frozen TerraMAE encoder and a lightweight CNN or linear probe. For soil texture prediction, where sand and silt fractions are predicted per pixel using POLARIS soil maps as ground truth, TerraMAE + CNN reaches MAE $I_j$3 with standard deviation $I_j$4, compared with $I_j$5 for the baseline MAE model and $I_j$6 for ResNet-50. The paper characterizes this as 6.6% lower error than ResNet-50 and 15.7% better than the baseline MAE, with more stable convergence and lower final errors (Faruk et al., 9 Aug 2025).

For crop type identification using the Cropland Data Layer with six major crop and land-cover classes and weighted loss to address class imbalance, TerraMAE + CNN achieves $I_j$7 top-1 accuracy and $I_j$8 mean IoU. The corresponding results are $I_j$9 and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$0 for ResNet-50, and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$1 and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$2 for the baseline MAE. The paper notes improved performance for both dominant and minority crops while also stating that very underrepresented classes remain difficult, which it frames as a general remote-sensing problem (Faruk et al., 9 Aug 2025).

For land cover classification using NLCD across California, Colorado, and Kansas, with classes including Shrub/Scrub, Herbaceous, Forest, Cropland, and minor cover types, TerraMAE + CNN attains $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$3 top-1 accuracy and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$4 mean IoU, compared with $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$5 and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$6 for ResNet-50, and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$7 and $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$8 for the baseline MAE. The paper highlights that these gains are obtained with a frozen TerraMAE encoder and without full-network fine-tuning (Faruk et al., 9 Aug 2025).

Task	Model	Result
Reconstruction	TerraMAE (SCI, MAE+SSIM+SID)	MAE $$SCI_{i, j} = 1 - \frac{|I_i - I_j|}{I_i + I_j + \epsilon}.$$9, SSIM $[0,1]$0
Reconstruction	Baseline MAE	MAE $[0,1]$1, SSIM $[0,1]$2
Soil texture	TerraMAE + CNN	MAE $[0,1]$3
Crop type	TerraMAE + CNN	$[0,1]$4 acc, $[0,1]$5 IoU
Land cover	TerraMAE + CNN	$[0,1]$6 acc, $[0,1]$7 IoU

The paper also reports zero-shot transfers to unseen geographies such as Colorado and Kansas with only negligible loss in performance, presenting this as evidence of spatial-spectral generalizability. At the same time, it identifies a computational trade-off: the loss enhancements add approximately $[0,1]$8 per-batch compute, although the paper states that training converges faster and produces richer features. This trade-off is relevant to deployment decisions in large-scale pretraining settings.

7. Position within hyperspectral Earth observation

TerraMAE is presented as a masked autoencoding framework specialized for hyperspectral Earth observation rather than a generic transformer applied to HSI. Its principal contributions, as stated in the paper, are learning statistically, spatially, and spectrally coherent representations; incorporating real-world reflectance structure through SCI grouping; optimizing reconstruction with a physically motivated composite loss; and achieving state-of-the-art results on key geospatial tasks relative to the reported baselines (Faruk et al., 9 Aug 2025).

Conceptually, the framework links three levels of design: data statistics, pretext objective, and downstream transfer. The data statistics enter through adaptive channel grouping; the pretext objective is expanded beyond per-pixel reconstruction; and transfer is evaluated through frozen-encoder performance on crop, land-cover, and soil tasks. This suggests a model of HSI pretraining in which representation quality depends not only on masking ratio or backbone capacity, but on whether the learning problem is formulated to preserve both spatial organization and spectral identity.

A common misconception in this area is that strong reconstruction error alone is sufficient to demonstrate useful hyperspectral representations. TerraMAE explicitly argues against that view by optimizing and reporting spatial and spectral fidelity measures alongside MAE, and by tying reconstruction behavior to downstream transfer. Another potential misconception is that fixed wavelength partitions are necessarily adequate for HSI masking. The reported superiority of SCI grouping over k-Means, HAC, and domain-informed splits indicates that adaptive, dataset-driven group structure can be more effective in this setting.

In that sense, TerraMAE occupies a specific methodological niche: self-supervised HSI pretraining that treats the spectral axis as an empirical structure to be modeled, rather than as a static ordering to be partitioned heuristically.