OlmoEarth v1.1: Multimodal Earth Vision

• OlmoEarth v1.1 is a family of Vision Transformer-based models that integrates optical, radar, and map data using self-supervised learning for efficient Earth observation.
• It employs innovative techniques such as modality-aware masking, latent masked image modeling, and band dropout to stabilize training and significantly reduce computational costs.
• Extensive evaluations show state-of-the-art performance across diverse tasks while achieving notable efficiency gains in GPU-hours and resource usage.

OlmoEarth v1.1 is a family of Vision Transformer (ViT)–based foundation models specifically designed for multimodal, spatio-temporal Earth observation data. It introduces a stable self-supervised learning regime in latent space, supports optical, radar, and high-quality map layers as input, and significantly reduces computational costs while maintaining state-of-the-art performance across a wide range of research and operational tasks. OlmoEarth v1.1 is fully open, with code, datasets, and pre-trained weights available for environmental, humanitarian, and research applications (Herzog et al., 17 Nov 2025, Tseng et al., 20 May 2026).

1. Architectural Advances and Design

OlmoEarth v1.1 extends the original ViT encoder–decoder paradigm for remote sensing by introducing several architectural modifications to optimize for efficiency and multimodal fusion. The Transformer backbone supports the ingestion of a diverse space-time "cube" that includes Sentinel-1 SAR, Sentinel-2 MSI, Landsat-8 OLI, and six map-derived layers (WorldCereal, WorldCover, OpenStreetMap, Cropland Data Layer, SRTM, Canopy Height). Each modality is tiled into patches at 10 m resolution; all patches are projected to a common $d$-dimensional token representation (Herzog et al., 17 Nov 2025).

Key innovations in v1.1 include:

• Collapsing all Sentinel-2 bands into a single bandset per time-step, reducing token count by approximately threefold and correspondingly reducing encoding multiply-accumulate operations (MACs) by 2.9$\times$ on S2 tasks (Tseng et al., 20 May 2026).
• Patch embedding via a two-stage nonlinear projection: first projecting patches to an intermediate dimensionality with a ReLU activation, then to the final embedding dimension. This stabilizes gradient norms and benefits fine-tuning (Tseng et al., 20 May 2026).
• Patch embeddings, 2D sinusoidal positional encodings, temporal encodings, and learned modality embeddings are summed for each token, preserving structure across space, time, and modalities.

Both encoder and decoder are implemented as ViT blocks. Encoder depths are model-size dependent (4/12/12/24 layers corresponding to Nano/Tiny/Base/Large), with 8–16 attention heads. The decoder cross-attends masked tokens for denoising and reconstruction tasks.

2. Self-Supervised Learning Formulation

OlmoEarth v1.1 is trained with a customized self-supervised objective tailored for the spatio-temporal, multimodal landscape of remote sensing data.

Modality-Aware Masking

Each input bandset in a training example is selectively assigned one of four roles:

• Not selected
• Encode only (visible to the encoder)
• Decode only (target for the decoder)
• Encode + Decode (masked targets)

Map layers are used exclusively as "decode only" for supervision and are never visible to the encoder. For temporal robustness, the model applies time-step masking with probability $p_t = 0.5$, removing entire sequences of data from the input for certain dates. This yields empirical gains over random token masking (Tseng et al., 20 May 2026).

Latent Masked Image Modeling (LatentMIM Lite)

The target encoder for masked patch prediction becomes a frozen random linear projection for each modality: $$T_m : \mathbb{R}^{p \times p \times c_m} \rightarrow \mathbb{R}^d$$. This construction is never trained, providing a stable reference for the decoder (Herzog et al., 17 Nov 2025).

Losses

Two principal losses are combined:

• Modality Patch Discrimination Loss $\mathcal{L}_{patch}$: For each masked token, the decoder's prediction is contrasted via an InfoNCE-style objective against all ground-truth same-modality tokens in the batch. For "map" modalities, negative tokens with cosine similarity $\geq 0.999$ are removed as "too hard" negatives (Tseng et al., 20 May 2026).
• Instance Contrastive Loss $\mathcal{L}_{inst}$: SimCLR-based contrastive loss is applied on mean-pooled encoder outputs from differently masked views of the same sample.

The final pretraining objective is:

$$\mathcal{L} = \mathcal{L}_{patch} + \lambda \mathcal{L}_{inst}$$

where the instance loss weight is $\lambda=0.05$ in v1.1 (Tseng et al., 20 May 2026).

Band Dropout

To compensate for any loss of spectral information from collapsing bandsets, input bands are randomly zeroed with probability $r \sim \text{Uniform}[0, 0.2]$ per forward pass in the encoder (target encoder always receives full input). This regularization is essential for maintaining transfer performance; ablations show dramatic performance drops when it is removed (Tseng et al., 20 May 2026).

3. Pretraining Protocol and Efficiency Improvements

OlmoEarth v1.1 processes 285,288 geodiverse samples, each a 2.56$\times$0 spatial tile with up to 12 monthly time-steps per modality from Jan 2016–Dec 2024. Data is uniformly sampled by OpenStreetMap category for broad coverage. All inputs are resampled to 10 m/pixel and missing values are masked (Herzog et al., 17 Nov 2025).

Significant efficiency improvements include:

• Reduction in GPU-hours: The Base model requires 1,763 GPU-hours for training, a 1.7$\times$1 reduction over v1 (2,989 GPU-hours) (Tseng et al., 20 May 2026).
• Reduction in MACs: Sentinel-2 encoding MACs decrease by 2.9$\times$2.
• Environmental Impact: Pretraining Nano, Tiny, and Base together uses 3,655 GPU-hours, compared to 5,987 for v1, reducing energy/carbon/water footprint by 69% (Tseng et al., 20 May 2026).

Optimizations such as replacing the patch embedding convolution with a reshape + nn.Linear layer and increasing micro-batch size for the contrastive loss from 32 to 64 further maximize utilization.

4. Benchmarking and Empirical Performance

OlmoEarth v1.1 is evaluated across 24 embedding (frozen) and 29 fine-tuning tasks, covering both standard research benchmarks and partner-driven real-world applications.

Evaluation Protocols

• Embedding (Frozen) Tasks: Performance is measured via $\times$3-nearest neighbors ($\times$4=20, cosine on pooled tokens) and linear probe (50 epochs, various learning rates) protocols.
• Fine-Tuning Tasks: Encoder is frozen for the first 20% of epochs, then full model is fine-tuned with AdamW and reduce-on-plateau scheduler.

Results

• OlmoEarth v1.1 achieves best performance on 15 of 24 embedding tasks and 19 of 29 fine-tuning tasks (Herzog et al., 17 Nov 2025).
• Average scores (frozen/finetuned): Nano (60.9/72.8), Tiny (62.1/77.0), Base (64.3/78.5). These closely match or slightly surpass v1, with improvements noted on tasks such as m-bigearthnet, and minor drops on specific cases like m-EuroSAT (Tseng et al., 20 May 2026).
• Ablations confirm all three v1.1 innovations (single bandset, band dropout, loss pruning) are individually necessary for Pareto-optimal performance.

Key tasks with strong performance improvements include Mangrove Watch (F1=98.1%, +2.8% vs. random forest baseline), Solar Farm segmentation (mIoU=86.7%), and reliable gains in multi-date crop type and flood mapping.

5. End-to-End Platform Integration

OlmoEarth v1.1 is incorporated into the open Helios platform, providing full-stack tooling for satellite data management, collaborative labeling, reproducible experimentation, and scalable inference (Herzog et al., 17 Nov 2025). Platform components include:

Component	Description
Data Management	H3-indexed ingestion via Google Earth Engine API
Labeling UI	Annotation for points, polygons, and temporal labels
Model Registry	Containerized evaluation, hyperparameter sweeps (Beaker)
Inference/Mapping	Batch/pipeline inference, map dashboards, GeoTIFF export

Partners such as AWF, IFPRI, Global Mangrove Watch, and the Global Ecosystems Atlas leverage these capabilities to iterate on data, annotation, and model retraining without direct GPU management.

6. Reproducibility, Open Resources, and Deployment

OlmoEarth v1.1 is fully open-sourced:

• Codebase for pretraining: https://github.com/allenai/olmoearth_pretrain
• Fine-tuning and evaluation: https://github.com/allenai/olmoearth_projects
• Pre-trained weights: Hosted on HuggingFace for Nano, Tiny, Base, and Large variants.
• Dataset: Released at https://huggingface.co/datasets/allenai/olmoearth_pretrain_dataset, with pre-aligned tiles and OpenStreetMap features.

Reproduction requires less than 10 minutes per task, following download and setup. Typical commands for training utilize a provided CLI and support configuration of masking, temporal dropout, band dropout, and other customizations as detailed in the released pseudocode and documentation (Tseng et al., 20 May 2026).

7. Practical Impact and Limitations

OlmoEarth v1.1 delivers significant reductions in operational costs, with the majority (98%) of large-scale mapping expense accruing at inference time. The computational optimizations of v1.1 yield 2–3$\times$5 cost reductions, improving accessibility for NGOs, researchers, and governments conducting high-resolution mapping. While overall performance is robust, there are small declines on select niche tasks (e.g., CropHarvest), but the Pareto frontier for accuracy versus compute is improved strictly over v1. All code, models, and data are open, further supporting broad adoption and reproducibility (Tseng et al., 20 May 2026, Herzog et al., 17 Nov 2025).