OlmoEarth Platform Overview

• OlmoEarth Platform is an open, end-to-end ecosystem for multimodal Earth observation, integrating advanced encoder–decoder Vision Transformers with scalable data ingestion.
• It employs a self-supervised LatentMIM Lite protocol and modular pipelines to process diverse remote-sensing data from Sentinel, Landsat, and multiple map products.
• Designed for robust spatiotemporal analysis, the platform supports real-time inference for applications such as land-cover mapping, hazard detection, and environmental monitoring.

The OlmoEarth Platform is a large-scale, open, end-to-end ecosystem for training, evaluating, and deploying multimodal Earth-observation foundation models. Centered around an encoder–decoder Vision Transformer architecture trained using a self-supervised “Latent MIM Lite” protocol, OlmoEarth is engineered for efficient and robust modeling of spatiotemporal, highly multimodal geospatial data. The platform integrates modular subsystems for data management, human-in-the-loop annotation, model pretraining and fine-tuning, and real-time/large-area inference, targeting a spectrum of use cases from land-cover mapping to hazard detection—especially in the environmental and humanitarian sectors (Herzog et al., 17 Nov 2025, Tseng et al., 20 May 2026).

1. Platform Architecture and System Components

OlmoEarth is architected as a modular platform encompassing four key subsystems [(Tseng et al., 20 May 2026), §4.1; (Herzog et al., 17 Nov 2025), §1]:

• Data Ingestion and Preprocessing: The ingestion pipeline supports three primary remote-sensing modalities (Sentinel-1 SAR, Sentinel-2 optical, Landsat-8 multispectral) and six auxiliary map products (WorldCover, Cropland Data Layer, SRTM elevation, canopy-height, WorldCereal, OpenStreetMap). Data is structured as tiles (e.g., 256×256 px or 2.56×2.56 km) linked to metadata (timestamp, geolocation) and managed in a global catalog with H3 indexing. Preprocessing includes radiometric calibration, cloud masking, co-registration, and time-series stacking.
• Patch Embedding and Tokenization: Preprocessed image cubes are partitioned into fixed-size patches and projected into a D-dimensional token space. Notably, v1.1 collapses each modality into a singular “bandset” per timestep (versus three previously for Sentinel-2), applying a learnable linear embedding (nn.Linear) instead of convolutional projections, resulting in a 1.23× throughput increase with identical weight mathematics (Appendix A, (Tseng et al., 20 May 2026)).
• Model Training Engine: OlmoEarth employs an encoder–decoder ViT trained via masked image modeling with a LatentMIM Lite protocol, using random or structured temporal/bandset masking, patch-level and instance-level (InfoNCE) contrastive losses, and negative sampling.
• Inference and Fine-Tuning Suite: Pretrained checkpoints (Nano, Tiny, Base) are served via a uniform inference engine supporting zero-shot embedding extraction (kNN, linear probing) and full fine-tuning. Downstream workflows—classification, segmentation, detection—are orchestrated via PyTorch scripts utilizing AdamW/LLRD optimizers, with fine-tuning recipes such as frozen-backbone then full adaptation [(Tseng et al., 20 May 2026), §4.1; (Herzog et al., 17 Nov 2025), §5].

2. Foundation Model Principles and Innovations

The core OlmoEarth foundation model is a spatiotemporal encoder–decoder Vision Transformer with extensible modality and temporal embeddings [(Herzog et al., 17 Nov 2025), §2]:

• Patch Embedding (“FlexiViT”): Supports variable patch sizes (1–8 px), adding 2D sin-cos positional, sinusoidal temporal, and learnable modality embeddings.
• Masking Strategy: Each input sample is subdivided into “bandsets” (e.g., Sentinel-2 10/20/60 m bands), and each is flagged as Not Selected, Encode-Only, Decode-Only, or Encode+Decode. Map modalities (e.g., OSM, WorldCover) are always Decode-Only, forcing nontrivial cross-modal reconstruction.
• Latent MIM Lite Loss: Rather than a learned target encoder, OlmoEarth uses a fixed, randomly initialized linear projection per modality to generate target representations, mitigating collapse and simplifying the loss computation:

$z = W_{\text{rand}} \cdot \text{patch\_pixels}$

The objective combines modality patch discrimination (local contrastive) and instance-level InfoNCE (global contrastive):

$$L_{\text{total}} = L_{\text{patch}} + \lambda L_{\text{instance}}$$

with λ=0.1 (v1), λ=0.05 (v1.1).

• v1.1 Model Refinements: Major advances in v1.1 include (1) single bandset grouping with random band dropout and nonlinear P×P×C→P×P×H→D projection (restoring spectral interaction and boosting kNN performance on benchmarks such as m-Eurosat), (2) structured temporal masking (all map tokens unmasked; temporal masking probability $p_t = 0.5$, uniformly improving mIoU), (3) refined loss with negative sample thresholding (cosine ≥ 0.999) for decode-only maps, halved instance loss weight, and larger micro-batches—improving training stability and reducing MACs by 2.9× (Sentinel-2), training GPU hours by 1.7× [(Tseng et al., 20 May 2026), §4.2].

3. Data, Labeling, and Workflow Pipelines

The data lifecycle in OlmoEarth is governed by scalable, reproducible protocols [(Herzog et al., 17 Nov 2025), §§3–4]:

• Sampling and Cataloging: Up to 12 modalities per tile, with systematic sampling across 120 OSM feature categories and ~285,000 global tile samples. Each tile maintains a full metadata entry (coords, time span, modality presence).
• Batching: Pretraining draws random tiles, time windows (2016–2024), patch sizes (1–8), crops, and sequence lengths (3–12 timesteps). All batch parameters are defined in open YAML config files (Tseng et al., 20 May 2026).
• Labeling and Annotation: Integrated web UI (“Helios”) supports point/polygon labels, multi-round review, audit logs, and active learning loops (model-inference proposals) with dataset versioning and export (TFRecords, COCO, GeoJSON).
• Fine-Tuning Pipelines: Recipe involves freezing the encoder for 20% of epochs (with only decoders trained), followed by joint training. Decoders (MLP, U-Net, Faster-RCNN) are auto-selected for task type. AdamW optimizers, LR plateaus, and LLRD are supported; fine-tuning distributed over partner research datasets and tasks [(Herzog et al., 17 Nov 2025), §5; (Tseng et al., 20 May 2026), §4.4].

4. Inference, APIs, and Integration

OlmoEarth’s deployment infrastructure is designed for low-latency real-time inference and bulk mapping at continental scale [(Herzog et al., 17 Nov 2025), §6]:

• API Endpoints: REST and gRPC serve /embed (tile embeddings), /infer/classify, /infer/segment, and /infer/detect with per-task outputs (class probabilities, segmentation masks, detection boxes).
• Performance: Single-tile inference latency is ≈100 ms (96×96×12 time-series on A100), with throughput up to 1,000 tiles/s; asynchronous mapping covers 10,000 km² in ≈20 min.
• Integration: Python SDK enables one-line execution for all core tasks and GIS plugins overlay maps in QGIS/ArcGIS. A CLI toolchain supports local testing and bulk cloud jobs.

5. Evaluation, Benchmarks, and Environmental Impact

Evaluation covers embedding-based tasks, full fine-tuning, and environmental footprints [(Herzog et al., 17 Nov 2025), §7; (Tseng et al., 20 May 2026), §4.3]:

• Embedding Evaluation: On 13 public benchmarks (m-BigEarthNet, m-So2Sat, m-Eurosat, PASTIS, MADOS), OlmoEarth Base achieves 67.8% average kNN/LP accuracy; Tiny 63.4%, Nano 61.6%. OlmoEarth outperforms 12 other models on 15/24 tasks.
• Fine-Tuning: Across 10 research and 19 partner tasks, OlmoEarth is top-performing on 5/10 research and 14/19 partner leaderboards. Representative results: m-so2sat (kNN 67.7% vs. next-best 60.5%), PASTIS mIoU (50.6 vs. 44.7), AWF (87.3% vs. prior 78.0%), Nandi crop-type (82.2% vs. prior 70.6%).
• Efficiency Gains (v1.1): MACs per example (Base): 2.74×10¹¹ (v1) → 9.46×10¹⁰ (v1.1); 2.9× reduction. Training GPU hours (Base): 2,989 → 1,763 (1.7× reduction), enabling broader accessibility.
• Environmental Metrics: Total training: ~4.3 MWh (1.7 tCO₂e, 6.7 kL water); fine-tuning: ~1.2 MWh (0.5 tCO₂e, 1.8 kL water) [(Herzog et al., 17 Nov 2025), §7.3; (Tseng et al., 20 May 2026)].

6. Applications and Case Studies

OlmoEarth’s universal representations are suited to a diverse range of scientific, conservation, and operational use cases [(Herzog et al., 17 Nov 2025), §8; (Tseng et al., 20 May 2026), §4.5]:

• Land Cover / Crop Classification: Models (zero-shot or fine-tuned) support maps at global (m-BigEarthNet, m-Eurosat), regional (So2Sat, CropHarvest, BreizhCrops), and high-resolution (AWF, Nandi) scales.
• Temporal and Change Analysis: Temporal masking in pretraining enhances deforestation detection and live fuel moisture estimation (Forest Loss Driver, Globe-LFMC).
• Hazard Monitoring: SAR-optical fusion learns robust flood detection (Sen1Floods11, Rapid Inundation).
• Infrastructure & Marine Mapping: Vessel, platform, solar-farm detection and regression using multimodal imagery and map fusion.
• Partner Deployments: Mangrove Watch (98.1% F1, up from ~95.3%), AWF (86.0% vs. 78.0%), Nandi crop-types (82.2% with field surveys), marine infrastructure (F1=89.3%), solar-farm segmentation (mIoU=86.7%).
• NGO / Non-Profit Integration: Data-to-insights cycles require no local GPU infrastructure; annotation to map publishing typically completes in under 24 hours [(Herzog et al., 17 Nov 2025), §9].

7. Open Source and Reproducibility

All code, model weights, and pretraining manifests for OlmoEarth are released under the Allen Institute OlmoEarth license [(Tseng et al., 20 May 2026), §4.4; (Herzog et al., 17 Nov 2025), abstract].

Component	Repository Path	Description
Data loaders	src/data	Multi-modal ingestion, cloud masking, time-series assembly
Model definition	src/models	ViT backbone, nonlinear projection, masking scheduler
Loss functions	src/loss	Patch discrimination, InfoNCE instance, negative thresholding
Training scripts	train_pretrain.py	AdamW, LR scheduler, mixed-precision
Fine-tuning scripts	src/finetune	kNN, linear probe, full fine-tune orchestration
Config files	configs/*.yaml	Hyperparameters, batch sizes, dropout rates, negative thresholds

All platform improvements (e.g., single bandset grouping, two-stage nonlinear projection, random band dropout, updated masking/loss regimes) are fully reproducible via public scripts and configuration templates (Tseng et al., 20 May 2026).