OlmoEarth: Multimodal EO Foundation Model

• OlmoEarth is a multimodal, spatio-temporal model designed for global-scale Earth observation, integrating satellite and ancillary data for comprehensive geospatial analysis.
• It employs a unified Vision Transformer architecture with FlexiViT patch embedding and novel self-supervised learning techniques to optimize spatial, temporal, and modality feature extraction.
• The platform offers scalable, efficient deployment for tasks like land-cover mapping, change detection, and cross-modal retrieval, achieving state-of-the-art benchmarks in both research and operational settings.

OlmoEarth is a multimodal, spatio-temporal foundation model and platform for Earth observation (EO), designed to address the domain-specific demands of global-scale remote sensing. It integrates multiple satellite and ancillary data modalities, employs novel self-supervised representation learning, and is paired with an end-to-end deployment platform for applied geospatial inference tasks. Released by the Allen Institute for AI, OlmoEarth has demonstrated state-of-the-art (SOTA) performance on research, benchmarking, and operational scenarios for land-cover mapping, change detection, and cross-modal retrieval in the EO domain (Herzog et al., 17 Nov 2025, Tseng et al., 20 May 2026).

1. Model Design and Architecture

OlmoEarth is structured around a unified encoder–decoder Vision Transformer (ViT) architecture, optimized for multimodal and multi-temporal remote sensing inputs. The architecture components are:

• Inputs: Each “instance” is a spatio-temporal data “cube” with up to 12 monthly timesteps comprising:
  • Sentinel-1 Synthetic Aperture Radar (SAR)
  • Sentinel-2 MultiSpectral Instrument (MSI), partitioned by spatial resolution
  • Landsat-8 MSI, divided by native bands
  • Geophysical and derived map products (WorldCereal, WorldCover, SRTM, canopy height, Cropland Data Layer, OpenStreetMap)
• Patch Embedding: Utilizing a “FlexiViT” scheme, each patch—ranging from 1×1 to 8×8 pixels—is linearly projected into a D-dimensional token.
• Embedding Strategy:
  • 2D spatial sinusoidal positional encoding
  • Sinusoidal temporal encoding
  • Learnable modality embeddings
• Encoder: Standard ViT stack self-attends jointly across spatial, temporal, and modality axes. Four model sizes are trained (Nano: 1.4M, Tiny: 6.2M, Base: 90M, Large: 300M parameters).
• Decoder: Four-layer Transformer, reconstructing latent representations for masked tokens via cross-attention to the encoder’s visible tokens (Herzog et al., 17 Nov 2025).

The v1.1 release introduces key optimizations:

• Collapsing multiple “bandsets” per modality (e.g., for Sentinel-2) into a single bandset, reducing token count and memory footprint
• Replacing convolutional patch embedders with reshape + linear projection for higher throughput
• Down-weighted instance-level contrastive loss for improved training stability (Tseng et al., 20 May 2026)

2. Self-Supervised Learning Objective and Masking

OlmoEarth trains under a composite self-supervised loss, “Latent MIM Lite,” designed for multimodal EO data:

• Modality Patch Discrimination Loss (Local): For each masked token, contrast latent predictions $\hat{y}_i$ with frozen targets $z_i$, restricting negative samples to the same modality bandset. The formulation is:

$$L_{\text{patch}} = -\sum_{i\in M} \log \frac{\exp(\mathrm{sim}(\hat{y}_i,z_i)/\tau)}{\sum_{j\in B(m)} \exp(\mathrm{sim}(\hat{y}_i,z_j)/\tau)}$$

where $\mathrm{sim}(u, v) = u \cdot v / \|u\|\|v\|$, and $B(m)$ indexes bandset $m$ negatives.

• Instance-Level Contrastive Loss (Global): Two randomly masked views of each data instance are encoded, pooling encoder outputs $u'$ and $u''$ for instance discrimination via InfoNCE:

$$L_{\text{inst}} = -\sum_{x} \log \frac{\exp(\mathrm{sim}(u'_{x},u''_{x})/\tau')}{\sum_{y\neq x} \exp(\mathrm{sim}(u'_{x},u''_{y})/\tau')}$$

• Masking Strategy: Each pass randomly samples “encode” and “decode” bandsets, yielding four modes per band: not selected, encode-only, decode-only, or encode+decode. A second independent mask is used for contrastive pairs.

In v1.1, masking is further structured by always unmasking ancillary map-only modalities and, with probability $p_t = 0.5$, masking entire timesteps to enforce temporal interpolation. Random band dropout is introduced per forward pass to bolster spectral representation (Herzog et al., 17 Nov 2025, Tseng et al., 20 May 2026).

3. Training Data, Preprocessing, and Optimization

OlmoEarth pre-training utilizes a global-scale dataset with the following specifications:

• Samples: 285,288 tiles, each 2.56 km × 2.56 km, spanning December 2015–2024 and up to 12 monthly observations
• Modalities: All imagery resampled to 10 m/pixel
• OpenStreetMap: 120 feature categories, up to 10,000 tiles per category
• Data Treatment: Bandsets are resampled, and geography is globally representative.

Optimization and hyperparameters:

• Optimizer: AdamW with initial learning rate $z_i$0, weight decay 0.02
• Batch Size: 512 (instance contrast micro-batch size 32; v1.1 increases to 64 due to reduced memory use)
• Schedule: 8,000-step linear warm-up, cosine decay
• Pretraining Steps: 667,200, covering $z_i$1 tokens per model (Herzog et al., 17 Nov 2025).

4. Efficiency Improvements and v1.1 Innovations

OlmoEarth v1.1 implements architectural and algorithmic refinements for cost-effective training and inference:

• Single-Bandset Tokenization: Bandset collapse reduces token count per input by 3×, cutting inference cost.
• Patch Projection: Introduces a non-linear, two-stage projection (pixels → $z_i$2 via ReLU; then $z_i$3) to maintain expressivity, especially with random band dropout.
• Contrastive Loss Filtering: Excludes “trivial” contrastive negatives (cosine similarity $z_i$4, decode-only target) to avoid collapse.
• Downward Loss Scaling: Instance loss weighting is reduced from $z_i$5 for improved stability.

Empirically, v1.1 Base model pre-training time drops from 2,989 to 1,763 GPU-hours (1.7× reduction). Inference multiply-accumulate operations (MACs) on Sentinel-2 tasks drop from 274G to 94.65G, i.e., 2.9× speedup at similar or better average performance (Tseng et al., 20 May 2026).

5. Benchmarks, Comparative Performance, and Trade-offs

OlmoEarth has been benchmarked against 12 prior foundation models under research and real-world tasks, with detailed outcomes:

• Embedding Benchmarks (Frozen kNN/Linear Probe):
  • Outperforms all competitors in 15/24 tasks (e.g., m-eurosat top kNN: 94.7%, best among all)
  • On m-bigearthnet (μF1): Nano 59.5, Tiny 59.4, Base 62.4, Large 62.0 (top score: 62.4)
• Full Fine-tuning:
  • SOTA on 19/29 tasks (m-eurosat Base: 98.7% accuracy; AWF land-cover Kenya: 87%)
  • Draught-induced fuel moisture regression (LFMC MSE): 17.9 vs. CopernicusFM 24.5 (lower better)
• Efficiency Pareto: OlmoEarth establishes a new frontier in MACs/accuracy trade-off, especially with v1.1 upgrades.

Modest regressions (1–3 percentage points) occur on spectral-rich tasks like m-eurosat after token count collapse, but are offset by improved scaling and inference (Herzog et al., 17 Nov 2025, Tseng et al., 20 May 2026).

6. Platform, Deployment, and Real-World Use Cases

OlmoEarth Platform is an end-to-end toolkit that integrates raw EO data alignment, human-in-the-loop annotation (web UI for points, polygons, consensus review), model fine-tuning (recipes for classification/segmentation/detection), and scalable inference (GeoTIFF exports). It supports SOTA workflows for research and operational partners:

• Case Studies:
  • Global Mangrove Watch: jump from baseline F1 95.3 to OlmoEarth F1 98.1; ongoing monthly inference
  • Global Ecosystem Atlas: fine-tuning with 1,500 labels delivers highest known accuracy for the region, supporting iterative labeling (Herzog et al., 17 Nov 2025)

All code, pre-trained weights, and datasets are open source (github.com/allenai/olmoearth_pretrain, HuggingFace models and datasets), and an online demo is provided.

7. Impact, Environmental Considerations, and Future Directions

OlmoEarth demonstrates that explicit architectural design and domain-tailored learning strategies enable efficient, accurate, and accessible EO foundation models. Environmental impact estimates for large-scale pre-training are reported at 4,307 kWh (1.72 tCO₂eq, 6.67 kL water) per full run (Herzog et al., 17 Nov 2025), and subsequent v1.1 optimizations further reduce resource requirements by 1.7× for training and 2.9× for inference (Tseng et al., 20 May 2026).

A plausible implication is that large-scale, open, and multimodal EO model development can be democratized for global non-profits and academic users, advancing the capabilities for geospatial AI with sustainable budgets and modest hardware. The trade-off paradigm introduced by v1.1 demonstrates that judicious architecture and algorithm selection can meaningfully shift the performance vs. cost frontier for Earth observation at scale.