MMEarth: Multi-modal Geo Representation
- MMEarth is a global, geo-aligned, multi-modal Earth observation dataset comprising 1.2 million locations and 12 modalities, designed for cross-modal pretext supervision.
- MP-MAE employs a ConvNeXt V2 based encoder that reconstructs masked Sentinel-2 patches and leverages cross-modal prediction tasks to force semantic abstraction.
- Empirical results on GEO-Bench datasets show that the multi-pretext approach enhances transfer performance, label efficiency, and robustness in few-shot settings.
MMEarth, commonly stylized with initial capitals in the original publication, is a global, geo-aligned, multi-modal Earth observation dataset paired with a pretraining framework, MP-MAE, for geospatial representation learning. The dataset comprises 1.2 million locations with 12 modalities and 46 total bands per location, and pairs pixel-level rasters and image-level metadata automatically by geographic coordinates and time. MP-MAE is a fully convolutional, ConvNeXt V2–based masked autoencoding approach that pretrains a single encoder on masked reconstruction of Sentinel-2 optical imagery together with cross-modal reconstruction and prediction tasks, with the stated goal of learning general-purpose representations for optical satellite images under severe label scarcity (Nedungadi et al., 2024).
1. Definition, motivation, and problem setting
MMEarth was introduced in response to a structural asymmetry in Earth observation. The volume of unlabelled EO data is very large, but many downstream applications still lack labelled training data. At the same time, EO differs from natural-image corpora in that multi-modal data from different sensors and products can be aligned automatically at virtually zero human labeling cost through shared geography and timestamps. MMEarth operationalizes that opportunity by constructing a large global corpus in which heterogeneous modalities are co-registered and temporally paired, then using those modalities as pretext targets rather than as mandatory inference-time inputs (Nedungadi et al., 2024).
This distinction is central to the design. MMEarth is not merely a dataset, and MP-MAE is not merely a masked autoencoder trained on Sentinel-2. The encoder is fed only Sentinel-2 visible patches during pretraining, but it is required to support reconstruction or prediction across multiple pixel-level and image-level targets. In the authors’ framing, this forces semantic abstraction beyond pixel-level textures, because several targets—such as climate statistics, biome class, ecoregion, geolocation, or month—cannot be inferred from local spectral texture alone in a trivial way (Nedungadi et al., 2024).
A common misunderstanding is to treat MMEarth as an any-to-any multimodal inference system. The original formulation is narrower and more specialized: it uses other modalities only as pretext supervision while specializing the encoder to Sentinel-2 at inference time. The paper explicitly contrasts this specialization with more general multimodal autoencoders such as MultiMAE and 4M, and reports that specialization to one input modality can outperform multi-input training for transfer in this setting (Nedungadi et al., 2024).
2. Dataset construction, scale, and modality structure
The dataset is global in coverage and biome-balanced in sampling. Locations are sampled uniformly across 14 biomes, and within each biome sampling is stratified by ecoregion area. For each location, time-critical modalities are collected around the Sentinel-2 observation date, and all pixel-level rasters are reprojected and resampled onto the 10 m Sentinel-2 grid to yield a harmonized data cube. The corpus covers four years, 2017–2020, and the full release occupies approximately 639 GB in gzip-compressed HDF5 format (Nedungadi et al., 2024).
Each sample is centered on a tile of 128 by 128 pixels at 10 m resolution, corresponding to approximately 1.28 km by 1.28 km on the ground. The modalities are divided into pixel-level and image-level groups.
| Level | Modalities | Form |
|---|---|---|
| Pixel-level | Sentinel-2, Sentinel-1, ASTER GDEM, ETH-GCHM, Dynamic World, ESA WorldCover | 128×128 at 10 m |
| Image-level | Biome, Ecoregion, ERA5 temperature, ERA5 precipitation, Geolocation, Month | Scalar per tile |
The pixel-level stack includes Sentinel-2 optical imagery with 12 bands, Sentinel-1 SAR with 8 bands, ASTER GDEM elevation and slope, ETH-GCHM canopy height and uncertainty, Dynamic World land-cover labels, and ESA WorldCover labels. The image-level stack includes biome, ecoregion, ERA5 temperature statistics, ERA5 precipitation statistics, cyclic geolocation encodings, and cyclic month encodings. Altogether, the dataset contains 12 modalities and 46 total bands per location (Nedungadi et al., 2024).
Preprocessing is modality-aware. Sentinel-2 images are selected with a tile-level cloud filter of CLOUD_PIXEL_PERCENTAGE < 10%; images with missing data at orbit boundaries are discarded; all pixel-level data are harmonized to the 10 m Sentinel-2 grid with bilinear interpolation; Dynamic World is aggregated over the year by mode to reduce noise and NO_DATA artifacts; and per-modality standardization to zero mean and unit variance is applied during pretraining, with Sentinel-2 L1C and L2A normalized separately. Missing input pixels are replaced by zeros, corresponding to the mean after normalization (Nedungadi et al., 2024).
Two reduced “taster” subsets were also released for limited-compute settings: MMEarth100k, containing 100k locations at 128×128, and MMEarth64, containing all 1.2 million locations center-cropped to 64×64 (Nedungadi et al., 2024).
3. MP-MAE architecture and multi-pretext formulation
MP-MAE builds on the ConvNeXt V2 fully convolutional masked autoencoder. The encoder is sparse and operates on visible patches only. For EO imagery, the first layer is modified relative to the standard ConvNeXt V2 design: the original learned downsampling is replaced by an initial 3×3 stride-1 convolution, followed by a depth-wise convolutional downsampling layer, in order to preserve input resolution for medium-resolution satellite imagery. During pretraining, the input is a random 112×112 crop from the 128×128 tile, and a patch size of 16×16 yields a 7×7 token grid (Nedungadi et al., 2024).
The pretext inventory spans both pixel-level reconstruction and image-level prediction. Pixel-level continuous targets are supervised with mean squared error over masked patches; categorical pixel-level targets use cross-entropy over masked patches; image-level tasks are decoded from masked tokens via global average pooling over decoded non-visible patches, then a linear head. One shallow convolutional decoder is used per task, and a learnable mask token fills missing tokens so that decoders can operate as dense convolutions (Nedungadi et al., 2024).
For a pixel-level continuous modality , the reconstruction loss is given as
For pixel-level categorical targets such as Dynamic World, the paper gives
The global multi-task objective is explored in two variants: equal weighting and task-uncertainty weighting following Kendall et al. A representative uncertainty-weighted form reported in the paper is
Spatial and temporal metadata are encoded cyclically. Latitude and longitude are represented by sine and cosine components, and month is encoded as
An important methodological choice is that the same random mask is applied to all pixel-level modalities. This prevents shortcut solutions in which an unmasked auxiliary modality trivially reveals the masked content of another modality, and preserves the intended cross-modal difficulty of the pretext tasks (Nedungadi et al., 2024).
4. Training protocol and empirical performance
The reported pretraining recipe uses 200 epochs, effective batch size 4096, and base learning rate . Fine-tuning and linear probing are both run for 100 epochs with effective batch size 1024 and base learning rate . For semantic segmentation, the paper uses a two-phase schedule: 50 epochs with a frozen encoder and trainable U-Net decoder, followed by 150 epochs of joint fine-tuning, with batch size 32 and base learning rate 0.01 (Nedungadi et al., 2024).
The principal downstream evaluation is on GEO-Bench Sentinel-2 subsets: BigEarth20k, So2Sat20k, EuroSat16k, Cashew1k, and SAcrop3k. The paper first shows that domain-specific pretraining on Sentinel-2 already improves substantially over ImageNet-pretrained ConvNeXt V2 MAE. For example, Sentinel-2 multispectral pretraining yields BigEarth20k FT/LP of 78.6 / 64.5 versus 72.9 / 57.2 for ImageNet MAE, and Cashew1k FT of 84.5 versus 77.5 (Nedungadi et al., 2024).
The full multi-pretext MMEarth model improves classification further. Relative to Sentinel-2-only pretraining, BigEarth20k LP rises from 64.5 to 68.2, So2Sat20k LP rises from 28.6 to 44.2, and So2Sat20k FT rises from 46.2 to 55.9. The abstract summarizes these as approximately +4 percentage points on BigEarth linear probing and approximately +16 percentage points on So2Sat linear probing (Nedungadi et al., 2024).
The segmentation results are more nuanced. Full MMEarth reaches Cashew1k FT 81.4 and SAcrop3k FT 34.5. This is better than ImageNet MAE, but not uniformly better than Sentinel-2-only multispectral pretraining, which attains 84.5 on Cashew1k and 33.3 on SAcrop3k. The paper explicitly notes that pixel-level extra tasks are helpful but not uniformly superior for segmentation. This suggests that the strongest benefits of the multi-pretext design appear in linear probing, label efficiency, and parameter efficiency rather than in every dense-prediction setting (Nedungadi et al., 2024).
The few-shot and scaling analyses reinforce that reading. In extreme low-label settings, MMEarth consistently outperforms Sentinel-2-only and ImageNet MAE baselines on BigEarth and So2Sat. The MP-MAE Atto encoder, with about 3.7M parameters and 200 pretraining epochs, outperforms ConvNeXt V2 Tiny, with about 28M parameters and 800 ImageNet pretraining epochs, in few-shot linear probing. The taster subset MMEarth64 remains highly competitive, which the authors interpret as evidence that diversity and sample count matter more than larger receptive fields in this setup (Nedungadi et al., 2024).
5. Position within geospatial foundation-model research
Subsequent work has treated MMEarth as an important reference point for multimodal EO pretraining. A 2025 review of AI-driven quantitative remote sensing inversion lists mmEarth alongside SatMAE and GFM as a representative foundation model that leverages large-scale self-supervised pretraining and multimodal fusion for quantitative inversion, and characterizes it as a model family designed for pixel-level regression under heterogeneous inputs and domain shifts (Yu et al., 11 Jul 2025). That review also emphasizes a broader limitation shared by current EO foundation models, including mmEarth: limited physical interpretability, underdeveloped uncertainty quantification, and non-standardized fine-tuning for regression (Yu et al., 11 Jul 2025).
GeoMeld makes the lineage explicit. Its sourcing strategy uses 1.2 million centroids from MMEarth, and its authors state that mmEarth’s MP-MAE objective established cross-modal reconstruction as an effective pretraining signal. GeoMeld extends that paradigm by scaling to approximately 2.5 million spatially aligned samples, adding semantically grounded verified captions, and combining MP-MAE with JEPA and caption–vision contrastive alignment (Hasan et al., 12 Apr 2026). In that sense, GeoMeld can be read as a semantic and alignment-centric extension of the MMEarth recipe rather than a wholesale departure from it.
MMEarth-Bench, in turn, uses MPMAE pretrained on MMEarth as one of its baseline encoders and tests the broader transfer hypothesis under global environmental tasks with explicit geographic out-of-distribution splits. Its findings are mixed: multimodal pretraining tends to improve robustness in limited-data regimes, but geographic generalization remains poor. The benchmark’s proposed TTT-MMR method is framed partly as a response to that weakness, using all modalities available at test time as auxiliary reconstruction targets for model-agnostic adaptation (Gordon et al., 6 Feb 2026). This later result is significant because it tempers any simple narrative in which multimodal pretraining alone solves domain shift at global scale.
6. Resources, limitations, and enduring significance
MMEarth was released with a project page, dataset construction code, and MP-MAE training and evaluation code, and the dataset itself is released under CC BY 4.0. The paper also provides concrete practical guidance: use the full dataset when possible, use MMEarth64 when compute is limited, and consider updating the first convolutional layer during linear probing when downstream band sets differ from the pretraining band set, as in So2Sat’s 10-band configuration (Nedungadi et al., 2024).
The limitations are equally explicit. Evaluation is restricted to five GEO-Bench datasets; broader geographic and task coverage would further validate generalization. Pretraining lasts 200 epochs, which is shorter than more typical MAE schedules of 800–1600 epochs. Robustness to RGB-only inputs is not explicitly enforced. The dataset’s biome-balanced sampling emphasizes the natural world, so built-up areas are present but less emphasized. Mixed L1C/L2A Sentinel-2 inputs are unavoidable because L2A was not globally available before November 2018, and residual cloud pixels may remain because filtering is performed at the original 100 km × 100 km tile level rather than through per-sample cloud masks (Nedungadi et al., 2024).
Even with those caveats, MMEarth occupies a clear place in the evolution of geospatial foundation models. It demonstrated at large scale that automatically aligned EO modalities can serve as a dense source of pretext supervision, that cross-modal targets substantially improve linear-probe transfer and label efficiency for Sentinel-2 encoders, and that multimodal EO pretraining can be competitive with or superior to larger natural-image-pretrained baselines in downstream remote-sensing tasks (Nedungadi et al., 2024). A plausible implication is that its lasting contribution is methodological as much as empirical: it helped define cross-modal masked pretraining as a canonical design axis for Earth observation representation learning, and later systems have largely extended that axis rather than replacing it (Hasan et al., 12 Apr 2026).