---
title: 'OlmoEarth: Multimodal EO Foundation Model'
url: https://www.emergentmind.com/topics/olmoearth
type: topic
---

# OlmoEarth: Multimodal EO Foundation Model

OlmoEarth is a multimodal, spatio-temporal foundation model and platform for Earth observation (EO), designed to address the domain-specific demands of global-scale remote sensing. It integrates multiple satellite and ancillary data modalities, employs novel self-supervised representation learning, and is paired with an end-to-end deployment platform for applied geospatial inference tasks. Released by the Allen Institute for AI, OlmoEarth has demonstrated state-of-the-art (SOTA) performance on research, benchmarking, and operational scenarios for land-cover mapping, change detection, and cross-modal retrieval in the EO domain [2511.13655][2605.20804].

## 1. Model Design and Architecture

OlmoEarth is structured around a unified encoder–decoder Vision Transformer (ViT) architecture, optimized for multimodal and multi-temporal remote sensing inputs. The architecture components are:

- **Inputs:** Each “instance” is a spatio-temporal data “cube” with up to 12 monthly timesteps comprising:
  - Sentinel-1 Synthetic Aperture Radar (SAR)
  - Sentinel-2 MultiSpectral Instrument (MSI), partitioned by spatial resolution
  - Landsat-8 MSI, divided by native bands
  - Geophysical and derived map products (WorldCereal, WorldCover, SRTM, canopy height, Cropland Data Layer, OpenStreetMap)
- **Patch Embedding:** Utilizing a “FlexiViT” scheme, each patch—ranging from 1×1 to 8×8 pixels—is linearly projected into a D-dimensional token.
- **Embedding Strategy:**
  - 2D spatial sinusoidal positional encoding
  - Sinusoidal temporal encoding
  - Learnable modality embeddings
- **Encoder:** Standard ViT stack self-attends jointly across spatial, temporal, and modality axes. Four model sizes are trained (Nano: 1.4M, Tiny: 6.2M, Base: 90M, Large: 300M parameters).
- **Decoder:** Four-layer Transformer, reconstructing latent representations for masked tokens via cross-attention to the encoder’s visible tokens [2511.13655].

The v1.1 release introduces key optimizations:
- Collapsing multiple “bandsets” per modality (e.g., for Sentinel-2) into a single bandset, reducing token count and memory footprint
- Replacing convolutional patch embedders with reshape + linear projection for higher throughput
- Down-weighted instance-level contrastive loss for improved training stability [2605.20804]

## 2. Self-Supervised Learning Objective and Masking

OlmoEarth trains under a composite self-supervised loss, “Latent MIM Lite,” designed for multimodal EO data:

- **Modality Patch Discrimination Loss (Local):** For each masked token, contrast latent predictions $\hat{y}_i$ with frozen targets $z_i$, restricting negative samples to the same modality bandset. The formulation is:
  $$
  L_{\text{patch}} = -\sum_{i\in M} \log \frac{\exp(\mathrm{sim}(\hat{y}_i,z_i)/\tau)}{\sum_{j\in B(m)} \exp(\mathrm{sim}(\hat{y}_i,z_j)/\tau)}
  $$
  where $\mathrm{sim}(u, v) = u \cdot v / \|u\|\|v\|$, and $B(m)$ indexes bandset $m$ negatives.
- **Instance-Level Contrastive Loss (Global):** Two randomly masked views of each data instance are encoded, pooling encoder outputs $u'$ and $u''$ for instance discrimination via InfoNCE:
  $$
  L_{\text{inst}} = -\sum_{x} \log \frac{\exp(\mathrm{sim}(u'_{x},u''_{x})/\tau')}{\sum_{y\neq x} \exp(\mathrm{sim}(u'_{x},u''_{y})/\tau')}
  $$
- **Masking Strategy:** Each pass randomly samples “encode” and “decode” bandsets, yielding four modes per band: not selected, encode-only, decode-only, or encode+decode. A second independent mask is used for contrastive pairs.

In v1.1, masking is further structured by always unmasking ancillary map-only modalities and, with probability $p_t = 0.5$, masking entire timesteps to enforce temporal interpolation. Random band dropout is introduced per forward pass to bolster spectral representation [2511.13655][2605.20804].

## 3. Training Data, Preprocessing, and Optimization

OlmoEarth pre-training utilizes a global-scale dataset with the following specifications:

- **Samples:** 285,288 tiles, each 2.56 km × 2.56 km, spanning December 2015–2024 and up to 12 monthly observations
- **Modalities:** All imagery resampled to 10 m/pixel
- **OpenStreetMap:** 120 feature categories, up to 10,000 tiles per category
- **Data Treatment:** Bandsets are resampled, and geography is globally representative.

Optimization and hyperparameters:
- **Optimizer:** AdamW with initial learning rate $1\times10^{-4}$, weight decay 0.02
- **Batch Size:** 512 (instance contrast micro-batch size 32; v1.1 increases to 64 due to reduced memory use)
- **Schedule:** 8,000-step linear warm-up, cosine decay
- **Pretraining Steps:** 667,200, covering $\sim10^{11}$ tokens per model [2511.13655].

## 4. Efficiency Improvements and v1.1 Innovations

OlmoEarth v1.1 implements architectural and algorithmic refinements for cost-effective training and inference:

- **Single-Bandset Tokenization:** Bandset collapse reduces token count per input by 3×, cutting inference cost.
- **Patch Projection:** Introduces a non-linear, two-stage projection (pixels → $\mathbb{R}^{P\times P\times H}$ via ReLU; then $\mathbb{R}^{P\times P\times H}\rightarrow\mathbb{R}^D$) to maintain expressivity, especially with random band dropout.
- **Contrastive Loss Filtering:** Excludes “trivial” contrastive negatives (cosine similarity $\geq0.999$, decode-only target) to avoid collapse.
- **Downward Loss Scaling:** Instance loss weighting is reduced from $0.1\to0.05$ for improved stability.

Empirically, v1.1 Base model pre-training time drops from 2,989 to 1,763 GPU-hours (1.7× reduction). Inference multiply-accumulate operations (MACs) on Sentinel-2 tasks drop from 274G to 94.65G, i.e., 2.9× speedup at similar or better average performance [2605.20804].

## 5. Benchmarks, Comparative Performance, and Trade-offs

OlmoEarth has been benchmarked against 12 prior foundation models under research and real-world tasks, with detailed outcomes:

- **Embedding Benchmarks (Frozen kNN/Linear Probe):**
  - Outperforms all competitors in 15/24 tasks (e.g., m-eurosat top kNN: 94.7%, best among all)
  - On m-bigearthnet (μF1): Nano 59.5, Tiny 59.4, Base 62.4, Large 62.0 (top score: 62.4)
- **Full Fine-tuning:**
  - SOTA on 19/29 tasks (m-eurosat Base: 98.7% accuracy; AWF land-cover Kenya: 87%)
  - Draught-induced fuel moisture regression (LFMC MSE): 17.9 vs. CopernicusFM 24.5 (lower better)
- **Efficiency Pareto:** OlmoEarth establishes a new frontier in MACs/accuracy trade-off, especially with v1.1 upgrades.

Modest regressions (1–3 percentage points) occur on spectral-rich tasks like m-eurosat after token count collapse, but are offset by improved scaling and inference [2511.13655][2605.20804].

## 6. Platform, Deployment, and Real-World Use Cases

OlmoEarth Platform is an end-to-end toolkit that integrates raw EO data alignment, human-in-the-loop annotation (web UI for points, polygons, consensus review), model fine-tuning (recipes for classification/segmentation/detection), and scalable inference (GeoTIFF exports). It supports SOTA workflows for research and operational partners:

- **Case Studies:**
  - Global Mangrove Watch: jump from baseline F1 95.3 to OlmoEarth F1 98.1; ongoing monthly inference
  - Global Ecosystem Atlas: fine-tuning with 1,500 labels delivers highest known accuracy for the region, supporting iterative labeling [2511.13655]

All code, pre-trained weights, and datasets are open source (github.com/allenai/olmoearth_pretrain, HuggingFace models and datasets), and an online demo is provided.

## 7. Impact, Environmental Considerations, and Future Directions

OlmoEarth demonstrates that explicit architectural design and domain-tailored learning strategies enable efficient, accurate, and accessible EO foundation models. Environmental impact estimates for large-scale pre-training are reported at 4,307 kWh (1.72 tCO₂eq, 6.67 kL water) per full run [2511.13655], and subsequent v1.1 optimizations further reduce resource requirements by 1.7× for training and 2.9× for inference [2605.20804].

A plausible implication is that large-scale, open, and multimodal EO model development can be democratized for global non-profits and academic users, advancing the capabilities for geospatial AI with sustainable budgets and modest hardware. The trade-off paradigm introduced by v1.1 demonstrates that judicious architecture and algorithm selection can meaningfully shift the performance vs. cost frontier for Earth observation at scale.

Source: https://www.emergentmind.com/topics/olmoearth