Foundation Model Evaluation Benchmark

• Foundation model evaluation benchmark is a systematic protocol combining datasets, controlled splits, and metrics to assess model performance.
• It enforces strict region and sensor separation to prevent data leakage and ensure realistic out-of-distribution evaluation.
• Comprehensive metrics like OA, AA, F1-score, and Kappa quantify model strengths, architectural impacts, and operational generalization.

A foundation model evaluation benchmark is a systematically designed protocol, dataset suite, and set of metrics established to rigorously measure and compare the capabilities, generalization, and operational value of large pre-trained models across representative downstream tasks. Benchmarks in this category—spanning vision, language, multimodal, and scientific domains—are engineered to enable robust, fair, and reproducible comparisons, rigorously control for data leakage and task contamination, and isolate strengths and weaknesses in model architectures and pretraining strategies.

1. Motivations and Challenges in Benchmarking Foundation Models

Foundation models (FMs) are expected to generalize across diverse tasks and domains, making comprehensive, cross-task evaluation critical for both scientific progress and real-world deployment. However, the pace of FM development (e.g., in Earth observation, protein science, language, biometrics, clinical medicine) presents challenges: heterogeneity in datasets and tasks, lack of standardized protocols, and difficulty in assessing transfer and out-of-distribution (OOD) generalization. Classical benchmarks often suffer from saturation (models achieving near-perfect scores), limited scope, or artificiality, which reduces discriminatory power and practical relevance.

Recent work in hyperspectral remote sensing has highlighted that without rigorous, cross-domain, and cross-sensor benchmarks tuned for realistic operational settings, reported performance can be misleading—particularly under OOD conditions where domain and sensor shifts dominate generalization gaps (Elbarz et al., 13 Oct 2025).

2. Benchmark Components: Task, Dataset, and Region/Sensor Splitting

An effective FM benchmark must include:

• Representatively challenging downstream tasks: For instance, pixel-level binary classification of cereal vs. non-cereal using hyperspectral imagery represents both practical crop mapping needs and a canonical generalization problem, as used in "Benchmarking foundation models for hyperspectral image classification" (Elbarz et al., 13 Oct 2025).
• Explicitly partitioned datasets reflecting real deployment conditions: Training and test regions should differ in geography, climate, and sensor platform to enable true OOD evaluation. For instance, training in Aïn Orma (EnMAP sensor) and testing in Al Haouz (PRISMA sensor, spectrally harmonized) captures both geospatial and cross-sensor variability.
• Meticulous label acquisition and harmonization: Ground-truth is provided via expert-labeled polygons, cleaned and rasterized into binary masks with consistent class definitions.

Component	Realization in (Elbarz et al., 13 Oct 2025)
Training data	EnMAP imagery, Aïn Orma, ground-truth from field surveys/QField
Test data	PRISMA imagery, Al Haouz, spectrally matched to EnMAP
Label harmonization	Manual digitization, harmonization, binary mask creation
OOD control	No spatial overlap, strict split of regions and platforms

This rigorous region/sensor exclusion protocol is crucial to prevent spurious generalization from shared context or unintentional data leakage.

3. Benchmark Workflow and Model Evaluation Methodology

The benchmark protocol in (Elbarz et al., 13 Oct 2025) enforces strict, reproducible model comparison by:

• Freezing all pretrained backbones, and applying an identical upsampling and decoder structure, ensuring that differences are due to FM representation rather than head or training protocol.
• Spatial context for each pixel is controlled: $3 \times 3$ centered patches are the default, upsampled to $16 \times 16$ for ViT-family compatibility. An ablation paper reveals that larger patches increase risk of train/val overlap and information leakage, while not improving performance.
• Identical preprocessing, decoder, and optimizer settings (AdamW, weight decay, cosine annealing, cross-entropy loss) for all compared methods.
• No data leakage across any split.

Step	Protocol Element
Architecture	Pretrained backbone + fixed decoder + patch upsampler
Training	AdamW; Cosine annealing; Cross-entropy loss
Patch sizing	3x3 spatial window, upsampled for transformer compatibility
Fine-tuning	Only decoder/unfrozen head layers updated

This ensures a fair, apples-to-apples comparison.

4. Model Families and Comparative Performance

Three representative hyperspectral FMs are benchmarked:

• HyperSigma: Spectral-spatial fusion with sparse sample attention; HyperGlobal-450K pretraining.
• DOFA (Dynamic One-For-All): Wavelength-conditioned hypernetwork, cross-modal attention; multimodal pretraining (Sentinel-1/2, Gaofen, RGB, EnMAP).
• SpectralEarth ViT: Global, multi-temporal hyperspectral transformer with spectral adapters, pre-trained on 538k patches (415k locations).
• A compact "nano" SpectralEarth, trained from scratch, is included for architecture vs. pretraining analysis.

Model	OA (%)	AA (%)	F1 (Cereal)	Kappa
SpectralEarth ViT	93.5 ± 0.8	93.4 ± 0.8	90.0 ± 0.9	0.85
DOFA	62.6 ± 3.5	72.6 ± 2.4	62.4 ± 2.0	0.34
HyperSigma	34.5 ± 1.8	52.4 ± 1.3	48.8 ± 0.7	0.03
SpectralEarth-nano	91.0	not reported	not reported	n/a

SpectralEarth ViT vastly outperforms alternatives, with the compact variant nearly matching the performance of the full pre-trained base model—a finding that highlights the importance of architecture and spectral adaptation over sheer model size or pretraining scale alone.

5. Evaluation Metrics and Statistical Reporting

Performance is measured using:

• Overall Accuracy (OA):

$OA = \frac{\sum_{i=1}^C n_{ii}}{N}$

where $n_{ii}$ is the correct pixel count for class $i$, $C=2$, $N$ is total pixels.

• Average Accuracy (AA):

$AA = \frac{1}{C} \sum_{i=1}^{C} \frac{n_{ii}}{n_i}$

where $n_i$ is true pixel count for class $i$.

• F1-score (per class):

$$F1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

with standard definitions for Precision and Recall.

• Kappa coefficient: Measures agreement above chance.

All metrics are reported with mean ± standard deviation over multiple runs, reflecting robust and statistically meaningful evaluation.

6. Insights on Generalization and Benchmark Recommendations

The rigorous OOD evaluation protocol employed (training on EnMAP/region A, testing on PRISMA/region B, harmonized bands, no overlap) demonstrates that naive pretraining scale or model size does not guarantee transfer. Cross-region, cross-sensor splits with strict leakage control reveal marked differences in OOD robustness. Ablation studies indicate that excessive spatial context increases the risk of memorization without improving cross-domain accuracy, and that transformers' built-in receptive field suffices with small context windows.

A key recommendation is that architectural suitability and specialized spectral/spatial adaptation are central to generalization; blanket application of larger models or more data does not ensure operational performance. Benchmarks must stress realistic task heterogeneity—geographically distinct regions, heterogeneous sensors, and careful avoidance of training/test leakage.

7. Implications for Future Foundation Model Development and Benchmarking

The presented benchmark provides both a robust performance baseline and actionable methodological recommendations for FM development:

• Future models should target multimodal fusion (across spectral, SAR, thermal), improved masking/noise suppression, and architectural selection tailored for remote sensing OOD generalization.
• Controlled spatial context is recommended to balance spatial detail, reduce overfitting, and ensure true generalization.
• Benchmarking protocols must strictly enforce OOD evaluation (different region/sensor/season), controlling for all potential leaks.
• Models and code should be open and evaluation scripts reproducible, to foster transparent comparison and community-driven improvement.

This approach elevates hyperspectral FM benchmarking to a level necessary for reproducible, reliable operational deployment, offering a blueprint applicable to other domains where distribution gaps and operational variability dominate task performance (Elbarz et al., 13 Oct 2025).