EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation

Abstract: Current Earth observation benchmarks focus on measuring performance on diverse tasks and applications, typically measuring generalization in-distribution. But when models are deployed, they must generalize to myriad out-of-distribution scenarios, such as new time periods, geographies, scales, and sensors. We introduce EarthShift: the first public testbed for benchmarking robustness across multiple realistic distribution shifts encountered in remote sensing. EarthShift enables users to measure distributional robustness by comparing performance in- and out-of-distribution using datasets from paired datasets from different sources, temporal windows, geographic locations, and sensors. Our experiments on 8 geospatial foundation models (GFMs) and 11 tasks covering 5 shift types show that GFMs consistently perform 15-20% worse out-of-distribution on average regardless of model architecture, size, pre-training or fine-tuning strategy. We show that GFM robustness is similar to that of generic vision foundation models, and even fully-supervised models. This highlights a need for future research to strive for improvements in distributional robustness, not just performance, which can be benchmarked using EarthShift. We release our code and datasets to provide a testbed to guide future work to create foundation models that are robust and reliable in real-world applications. Code and data for EarthShift are available at: https://earthshift.github.io

• The paper introduces EarthShift, the first benchmark providing paired datasets for quantifying robustness under realistic distribution shifts in Earth observation.
• The evaluation shows significant performance drops (up to 60%) in out-of-distribution scenarios, with geospatial foundation models not consistently outperforming generic vision models.
• The study finds that full fine-tuning can worsen OOD generalization compared to frozen-backbone adaptation, highlighting the need for alternative robust adaptation methods.

EarthShift: Benchmarking Distributional Robustness in Remote Sensing

Motivation and Benchmark Design

EarthShift addresses the critical gap in evaluating distributional robustness for machine learning models deployed in Earth observation (EO), proposing the first comprehensive public testbed for quantifying robustness across realistic distribution shifts. Conventional benchmarks, such as those commonly used in SatML, focus on in-distribution (ID) performance and overlook the salient challenge of out-of-distribution (OOD) generalization—crucial for real-world EO applications, which routinely experience shifts in time, geography, scale, sensor modality, and source. EarthShift formally characterizes robustness via paired datasets that instantiate these shift types, systematically organizing 11 tasks over five shift categories (spatial resolution, temporal, geographic, sensor, source) (Figure 1).

Figure 1: Representative samples of EarthShift distribution shifts. ID denotes in-distribution data; OOD denotes out-of-distribution samples exhibiting a “shift”.

Effective robustness is measured using a regression-normalized framework [taori2020measuring], which decouples robustness from raw ID performance, avoiding the pitfalls of mere delta-based rankings and enabling meaningful model comparisons relative to task difficulty and shift severity.

Model Evaluation Protocol

A systematic evaluation was conducted across 13 model architectures, including 8 geospatial foundation models (GFMs)—spanning DOFA, CROMA, Clay, Prithvi V2.0, TerraFM, DINOv3, Galileo, and TerraMind—as well as generic vision foundation models (VFMs) and fully-supervised baselines (randomly initialized ResNet50/Vision Transformer). Tasks covered classification and semantic segmentation, under two transfer protocols: full fine-tuning and frozen-backbone adaptation. Robust channel and task adaptation, rigorous learning rate sweeps, and evaluation over 10,000+ experiments ensure statistically stable outcomes, with ID and OOD test sets derived from curated satellite data splits.

Empirical Results and Analysis

Distributional robustness analysis reveals several critical, quantified patterns:

• Performance degradation under distribution shift is ubiquitous. All models—regardless of architecture, size, pre-training corpus, or adaptation protocol—experience substantial OOD drops (15–20% on average), with variations up to 60% depending on shift severity (notably sensor shifts) (Figure 2).

  Figure 2: Effective distribution shift analysis for frozen backbone (left) and full fine-tuning (right): model-task datums relative to robustness baseline; dashed $y=x$ denotes perfect robustness.

• GFMs fail to confer robustness advantages relative to generic baselines. No GFM systematically outperforms VFMs (e.g., CLIP, ImageNet ResNet) or even randomly-initialized supervised models, challenging prevailing assumptions regarding domain-specific pre-training efficacy (Figure 3, Figure 4).

  Figure 3: Scatterplot of ID vs. OOD absolute performance for frozen backbone (a) and full fine-tuning (b). Shift type and task visualized by color and symbol.

  

  Figure 4: Model-type comparison of absolute OOD-ID performance gap per shift task.

• Temporal shifts are benign; sensor, geographic, and scale shifts induce sharp failures. All models are robust to seasonal and year-to-year temporal variations, but suffer sharply when exposed to new sensors, locations, and resolutions. Sensor shifts yield up to 60% drop in performance—especially between Sentinel-2 and Sentinel-1 bands.
• Fine-tuning does not mitigate robustness gaps; frozen backbones occasionally outperform full adaptation. In multiple shift scenarios, full fine-tuning exerts negative pressure on OOD generalization, in line with the notion that fine-tuning can distort pretrained representations and exacerbate overfitting to ID distributions (Kumar et al., 2022).

  Figure 5: Comparison of full fine-tuning vs. frozen backbone for absolute performance degradation (OOD-ID) per task.

• Model parameter count is not predictive of robustness. Across models ranging from 25M to 300M+ parameters, no correlation is observed between backbone size and OOD robustness delta, supporting that architectural scaling alone cannot resolve distributional brittleness.

Figure 6: Mean performance delta (OOD-ID score) versus model backbone size for frozen backbone adaptation.

Practical and Theoretical Implications

The EarthShift benchmark exposes fundamental limitations of current GFMs, implying that:

• Real-world deployment of EO models is severely hampered by lack of robustness. High average OOD degradation (20–25%) means predictions outside ID context are unreliable, necessitating continuous validation and retraining post-deployment.
• Domain-specific pre-training on EO datasets does not meaningfully enhance robustness. Architectural and pre-training advances (e.g., MAE, contrastive learning, multimodal integration) yield no tangible robustness gains compared to out-of-domain, generic vision pretraining, possibly due to inherent biases and lack of diversity within EO archives.
• Fine-tuning protocols merit reconsideration. Aggressive full parameter adaptation may undermine learned invariances; frozen-backbone transfer and alternative adaptation methods (e.g., test-time entropy minimization [wang2021tent]) require further exploration for robust deployment.
• Progress in synthetic shift augmentation and domain adaptation does not translate to natural distribution shift robustness. The severe brittleness observed is not reflected in synthetic benchmarks, reaffirming the need for benchmarks like EarthShift that stress natural shift scenarios.

Future Directions

The paper invites several research directions:

• Expansion of EarthShift with broader task coverage and new shift instantiations, supporting continual benchmarking as new models and datasets emerge.
• Exploration of architectures and training paradigms explicitly optimized for distributional robustness, including invariance learning, domain adversarial training, and automated OOD detection.
• Systematic investigation into the transferability of test-time adaptation and “living” models that update in response to evolving EO distributions.
• Integration of EarthShift into the evaluation pipeline for geospatial AI, raising the bar for robustness claims beyond ID-centric metrics.

Conclusion

EarthShift constitutes the first rigorous, comprehensive benchmark for distributional robustness in remote sensing, revealing endemic performance drops (~20% on average) under realistic shift conditions and refuting claims of robustness benefits from geospatial foundation model pre-training. The benchmark, codebase, and datasets are publicly released, providing a critical resource for the development of next-generation GeoAI models capable of reliable out-of-distribution generalization (2605.29330).