EarthShift: Benchmarking Robustness in Remote Sensing

• EarthShift is a benchmark for measuring robustness to real-world distribution shifts in remote sensing by comparing in-distribution vs. out-of-distribution performance.
• It assesses models using paired datasets across five natural shift types—temporal, geographic, scale, sensor, and source—to simulate realistic operational scenarios.
• Empirical findings show a consistent 15–20% performance drop under shifts, indicating that higher in-distribution accuracy does not ensure deployment reliability.

Searching arXiv for the specified paper and the cited effective robustness framework. EarthShift is a benchmark for measuring robustness to real-world distribution shifts in Earth observation. It was introduced as “the first public testbed for benchmarking robustness across multiple realistic distribution shifts encountered in remote sensing,” with the explicit aim of comparing in-distribution and out-of-distribution performance across paired datasets that differ in source, temporal window, geography, scale, and sensor (Doerksen et al., 28 May 2026). The benchmark addresses a central limitation of conventional Earth observation evaluation practice: most existing benchmarks measure generalization in-distribution, whereas deployed models must operate under shifts such as new time periods, new geographic regions, different ground sampling distances, distinct collection pipelines, and sensor changes. EarthShift therefore formalizes robustness as the performance gap between held-out test data from the training distribution and paired test data drawn from a shifted distribution, and uses that framework to benchmark modern geospatial foundation models, generic vision foundation models, and fully supervised baselines (Doerksen et al., 28 May 2026).

1. Motivation and problem setting

Remote-sensing models are typically evaluated on held-out samples drawn from the same regions, seasons, sensors, and resolutions used during training. EarthShift is motivated by the observation that such evaluation is insufficient for deployment, because real-world use requires extrapolation across new seasons, years, countries, continents, resolutions, and sensing modalities (Doerksen et al., 28 May 2026). The benchmark situates this problem in the broader context of distributional robustness, emphasizing that a model with strong in-distribution performance may nonetheless be brittle when exposed to realistic operational shifts.

The benchmark description highlights several concrete deployment contexts in which such brittleness is consequential, including crop monitoring, disaster response, and ecosystem mapping (Doerksen et al., 28 May 2026). It also notes that related applications have exhibited substantial degradation under shift, including “accuracy drops of 15–30 percent or more” and a “30 percent drop in building-damage detection across disaster sites” (Doerksen et al., 28 May 2026). Within that framing, EarthShift is designed to quantify what the paper terms the “robustness gap”: the decline from in-distribution to out-of-distribution performance measured under matched task definitions.

This suggests that EarthShift is not primarily a task benchmark in the conventional sense. Rather, it is an evaluation framework for stress-testing learned representations and transfer protocols under ecologically valid forms of covariate and domain shift. A plausible implication is that benchmark success on EarthShift is intended to be more predictive of operational reliability than benchmark success on standard in-distribution geospatial leaderboards.

2. Benchmark composition and shift taxonomy

EarthShift assembles 11 paired dataset test cases spanning scene classification, multi-label classification, and semantic segmentation, and organizes them around five “natural shift types” common in remote sensing (Doerksen et al., 28 May 2026). In every case, the source dataset defines the in-distribution regime and the paired dataset defines the out-of-distribution regime.

Shift type	Paired benchmark examples	Task family
Scale shift	RESISC45 ↔ UCMerced	Scene classification
Temporal shift	Fields of the World Window A ↔ Window B; Germany 2018 → 2019	Semantic segmentation
Geographic shift	Fields of the World Germany → Cambodia; Germany → Denmark	Semantic segmentation
Sensor shift	m-EuroSat RGB → RGE1; BigEarthNet v2 Sentinel-2 → Sentinel-1; Sen1Floods11 Sentinel-2 → Sentinel-1	Classification, multi-label classification, segmentation
Source shift	DeepGlobe RGB → DFC2022 RGB	Semantic segmentation

Scale shift is defined as variation in ground sampling distance under otherwise related semantic content. The benchmark instantiates this via RESISC45, with “0.2–30 m,” and UCMerced, with “0.3 m,” noting “45 vs. 21 classes, 19 shared classes” (Doerksen et al., 28 May 2026).

Temporal shift uses identical locations observed across seasons or years. EarthShift implements this with Fields of the World agricultural-field segmentation across “Window A ↔ Window B globally and South Africa” and “Germany 2018 → 2019” (Doerksen et al., 28 May 2026).

Geographic shift fixes the task while changing country or continent. The reported cases are “Germany → Cambodia” and “Germany → Denmark” for Fields of the World segmentation (Doerksen et al., 28 May 2026).

Sensor shift is defined over different sensors or channel subsets. The benchmark includes “m-EuroSat RGB (B02–B04) → RGE1 (B02–B03–B05) classification,” “BigEarthNet v2 multi-label: Sentinel-2 → Sentinel-1,” and “Sen1Floods11 segmentation: Sentinel-2 → Sentinel-1” (Doerksen et al., 28 May 2026).

Source shift captures visually similar data curated through different collection pipelines. EarthShift instantiates this as “DeepGlobe RGB semantic segmentation (DigitalGlobe satellite) → DFC2022 RGB segmentation (aerial)” (Doerksen et al., 28 May 2026).

The benchmark protocol is uniform across these cases: the model is fine-tuned only on the in-distribution dataset, specifically “Train + Val from source A,” and then evaluated on two held-out test sets, namely the held-out portion of source A as the in-distribution test set and the entirety of source B as the out-of-distribution test set (Doerksen et al., 28 May 2026). This paired construction is central because it enables direct estimation of the degradation associated with each shift type.

3. Evaluation protocol and robustness formalization

EarthShift uses task-appropriate metrics across its 11 test cases. Scene classification is evaluated with accuracy, multi-label classification with micro-F1, and semantic segmentation with mean intersection-over-union, denoted mIoU (Doerksen et al., 28 May 2026). For each model $f$ and each shift scenario, the benchmark records three quantities:

$$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$

$$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$

$$\Delta(f) = \mathrm{score}_{ID}(f) - \mathrm{score}_{OOD}(f)$$

These definitions make the robustness gap explicit. The benchmark does not collapse robustness into a single absolute out-of-distribution score, because raw out-of-distribution performance can be confounded by corresponding in-distribution strength. Instead, EarthShift adopts the “effective robustness” framework of Taori et al., defining

$$\rho(f) = \mathrm{score}_{OOD}(f) - \beta(\mathrm{score}_{ID}(f)),$$

where $\beta(\cdot)$ is a baseline mapping from in-distribution to expected out-of-distribution performance, fit by linear regression across all models for a given shift type:

$\beta(s_{ID}) = m \cdot s_{ID} + b.$

Under this definition, $\rho(f) > 0$ indicates that a model outperforms the out-of-distribution performance predicted by its in-distribution score, while $\rho(f) < 0$ indicates underperformance relative to that baseline (Doerksen et al., 28 May 2026). This construction is methodologically important because it distinguishes genuine robustness gains from mere improvements in in-distribution fitting.

All reported numbers are “means ± standard deviation over five random seeds,” and “confidence bands (±1 σ) around baseline slopes” are used to characterize the consistency of the fitted trends across tasks (Doerksen et al., 28 May 2026). The paper reports that the benchmark required “over 10,000 experiments” under two transfer protocols: full fine-tuning and frozen backbone with a trainable head (Doerksen et al., 28 May 2026). This scale of evaluation reflects a deliberate attempt to separate model-class effects, initialization effects, and adaptation-protocol effects from idiosyncratic task outcomes.

4. Model classes and transfer settings

EarthShift benchmarks 13 models. These are divided into three groups: eight geospatial foundation models, three generic vision foundation models, and two fully supervised models trained from scratch (Doerksen et al., 28 May 2026).

Model group	Models	Training characterization
Geospatial Foundation Models (GFMs)	DOFA, CROMA, Clay, Prithvi 2.0, TerraFM, DINOv3, Galileo, TerraMind	“self-supervised on multi-sensor EO data”
Generic Vision Foundation Models (VFMs)	ResNet-50, ViT-B/16, CLIP	“ImageNet supervised” for ResNet-50 and ViT-B/16; “contrastive” for CLIP
Fully supervised from scratch	ResNet-50-random, ViT-random	Random initialization

The benchmark evaluates each model under two transfer protocols: “full fine-tuning” and “frozen backbone + trainable head” (Doerksen et al., 28 May 2026). The benchmark documentation further states that new models can be incorporated by pointing to pre-trained checkpoints and wrapping them with EarthShift’s “1×1 channel adapters and classification/segmentation heads” (Doerksen et al., 28 May 2026). This indicates that EarthShift is intended not only as a fixed leaderboard but also as an extensible evaluation harness.

A central design implication is that EarthShift compares robustness across representation sources that differ substantially in pre-training domain, scale, and supervision. Because the benchmark includes domain-specific geospatial pre-training, generic natural-image pre-training, and random initialization, it directly tests whether remote-sensing-specific pre-training confers robustness advantages under realistic Earth observation shifts. The reported findings indicate that this expectation is not borne out straightforwardly.

5. Empirical findings

The principal quantitative result is that models “consistently perform 15–20% worse out-of-distribution on average regardless of model architecture, size, pre-training or fine-tuning strategy” (Doerksen et al., 28 May 2026). The detailed benchmark description likewise reports “Average performance drop $\Delta \simeq 15–20$ percent across all shifts and models” (Doerksen et al., 28 May 2026). This places EarthShift’s central claim on robustness degradation at the level of a systematic cross-model pattern rather than an isolated failure mode.

The shift-type breakdown under full fine-tuning is more differentiated. Temporal shift is associated with “slope $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$0, intercept $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$1, 2 percent mean $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$2,” and is summarized as indicating that “models largely robust to season/year changes” (Doerksen et al., 28 May 2026). Geographic shift yields “$$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$3, $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$4, $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$5 percent $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$6 when transferring Germany → Cambodia; smaller $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$7 Germany → Denmark” (Doerksen et al., 28 May 2026). Scale shift shows “$$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$8, $$\mathrm{score}_{ID}(f) = \text{performance on ID test set}$$9, $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$0 percent $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$1” (Doerksen et al., 28 May 2026). Sensor shift is reported as “$$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$2, $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$3, up to $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$4 percent $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$5 in some sensor-swap tasks” (Doerksen et al., 28 May 2026). Source shift has “$$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$6, $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$7, $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$8 percent $$\mathrm{score}_{OOD}(f) = \text{performance on OOD test set}$$9” (Doerksen et al., 28 May 2026).

These results establish a ranking of failure severity across shift types. The benchmark identifies temporal shifts as the least damaging and sensor and scale shifts as the most damaging, with source and some geographic transfers occupying an intermediate regime (Doerksen et al., 28 May 2026). The paper’s summary therefore states that “models are weakest under sensor and scale shifts (40–60 percent average $$\Delta(f) = \mathrm{score}_{ID}(f) - \mathrm{score}_{OOD}(f)$$0) and strongest under temporal shifts (≈2 percent $$\Delta(f) = \mathrm{score}_{ID}(f) - \mathrm{score}_{OOD}(f)$$1)” (Doerksen et al., 28 May 2026).

Effective robustness values cluster “around zero for nearly every model and shift,” meaning that no model class “consistently beat[s] the ID→OOD baseline” (Doerksen et al., 28 May 2026). This is an especially important negative result. It implies that higher in-distribution scores generally predict correspondingly higher out-of-distribution scores, but without evidence that any family of models delivers systematic robustness gains beyond that baseline expectation. EarthShift also reports that “GFM robustness is similar to that of generic vision foundation models, and even fully-supervised models” (Doerksen et al., 28 May 2026). In the detailed findings, this appears as “GFMs show no clear OOD advantage over ImageNet VFMs or even fully-supervised random-init models” (Doerksen et al., 28 May 2026).

Additional findings sharpen this conclusion. “Full fine-tuning sometimes worsens OOD robustness relative to frozen backbones” (Doerksen et al., 28 May 2026), and “larger models (e.g. Prithvi with 304 M params vs. ResNet-50 with 25 M) do not correlate with better robustness” (Doerksen et al., 28 May 2026). Taken together, these findings suggest that neither parameter count nor domain-specific pre-training nor aggressive adaptation is sufficient, by itself, to solve distribution shift in Earth observation.

6. Interpretation, implications, and common misconceptions

A common assumption in geospatial machine learning is that domain-specific pre-training on large Earth observation archives should confer meaningful robustness benefits under realistic deployment shift. EarthShift directly tests this assumption and reports the opposite pattern: “Current GFMs, despite domain-specific pre-training on vast archives (Sentinel-1, Sentinel-2, NAIP, Landsat, DEMs, etc.), do not yield superior distributional robustness compared to general-purpose ImageNet models or even models trained from scratch” (Doerksen et al., 28 May 2026). This does not imply that geospatial foundation models are unhelpful; it indicates specifically that their robustness under the benchmarked shifts is not clearly superior.

Another misconception is that better in-distribution performance should be treated as evidence of deployment readiness. EarthShift’s adoption of effective robustness is designed precisely to avoid that conflation. By tracking both absolute out-of-distribution performance and $$\Delta(f) = \mathrm{score}_{ID}(f) - \mathrm{score}_{OOD}(f)$$2, the benchmark distinguishes improvements that merely follow from higher in-distribution scores from improvements that exceed the expected in-distribution-to-out-of-distribution transfer relationship (Doerksen et al., 28 May 2026). A plausible implication is that leaderboard improvements that omit such normalization may overstate practical gains.

The reported transfer-protocol effects also complicate a common practice in foundation-model adaptation. EarthShift finds that “fine-tuning can improve in-distribution scores but often exacerbates OOD drops” (Doerksen et al., 28 May 2026). This indicates a tension between adaptation to source-domain idiosyncrasies and preservation of robust pre-trained features. The benchmark’s observation that frozen backbones can sometimes be more robust than fully fine-tuned models is consistent with that interpretation, though the benchmark does not claim a universal dominance of one protocol over the other.

Finally, the relative robustness to temporal shift should not be misread as a general solution to nonstationarity. EarthShift reports that models are “largely robust to season/year changes” in the specific benchmarked temporal settings, with a “2 percent mean $$\Delta(f) = \mathrm{score}_{ID}(f) - \mathrm{score}_{OOD}(f)$$3” (Doerksen et al., 28 May 2026). This is a benchmark-specific empirical statement rather than a blanket claim that seasonal or annual change is negligible in Earth observation.

7. Extensions, recommendations, and benchmark use

EarthShift is publicly released with “code and datasets” and provides “data splits and benchmarks” at earthshift.github.io (Doerksen et al., 28 May 2026). The benchmark description identifies several intended uses. Researchers can add new models by supplying pre-trained checkpoints and using EarthShift’s “1×1 channel adapters and classification/segmentation heads” (Doerksen et al., 28 May 2026). They can also extend the benchmark “to novel shifts by defining paired ID/OOD dataset splits in the same protocol” (Doerksen et al., 28 May 2026). The benchmark explicitly recommends tracking both absolute out-of-distribution performance and effective robustness $$\Delta(f) = \mathrm{score}_{ID}(f) - \mathrm{score}_{OOD}(f)$$4 “to prevent conflating in-distribution gains with true generalization” (Doerksen et al., 28 May 2026).

The paper also outlines directions for robustness research. It recommends that “pre-training objectives should explicitly target invariances across sensors, resolutions and geographies,” giving as examples “contrastive alignment of multi-sensor views” and “domain-adversarial learning” (Doerksen et al., 28 May 2026). It further recommends incorporating “synthetic or real OOD examples during pre-training,” including “self-supervised domain randomization” (Doerksen et al., 28 May 2026). Additional directions include “test-time adaptation without labels,” with examples such as “Tent entropy-minimization” and “batch-norm adaptation,” and designing “heads and architectures that preserve robust pre-trained features rather than overfit ID signals” (Doerksen et al., 28 May 2026).

These recommendations are presented as future research directions rather than validated benchmark conclusions. This suggests that EarthShift’s principal contribution is diagnostic: it establishes a rigorous and extensible testbed showing that current Earth observation models remain substantially vulnerable to realistic deployment shifts. In that sense, EarthShift reframes benchmark progress in geospatial AI away from raw in-distribution accuracy alone and toward explicit measurement of reliability under source, sensor, scale, geographic, and temporal mismatch (Doerksen et al., 28 May 2026).