MetaEarth-MM: Unified Multimodal Remote Sensing

• MetaEarth-MM is a unified generative foundation model for multimodal remote sensing that decouples scene content from modality-specific appearance.
• It employs continuous flow matching and a decoupled architecture with a DiT-based transformer to enhance modality-specific generation across RGB, SAR, NIR, PAN, and OSM.
• Empirical results on the EarthMM dataset show improved FID, CAS, and downstream task performance, validating its scene consistency and cross-modal translation capabilities.

Searching arXiv for MetaEarth-MM and directly related papers. MetaEarth-MM most commonly denotes a unified generative foundation model for multimodal remote sensing imagery that organizes generation around the underlying scene rather than learning isolated appearance-level mappings between modality pairs. In its explicit formulation, the model enables both paired joint generation and any-to-any translation across five modalities—RGB, SAR, NIR, PAN, and OSM—within a single framework by introducing a scene-centered joint modeling paradigm under flow matching (Yu et al., 19 May 2026). The name also appears in unrelated synthesized usages connected to wide-field mm/sub-mm transient astronomy and to a conceptual reframing of a next-generation mm/sub-mm array for Earth-analog detection, but those usages are not the official multimodal remote-sensing model introduced in the 2026 preprint (Koljonen et al., 16 Dec 2025, Saito et al., 2011). A closely related precursor, "MetaEarth: A Generative Foundation Model for Global-Scale Remote Sensing Image Generation," is optical-only and does not introduce an official multimodal variant called MetaEarth-MM (Yu et al., 2024).

1. Definition and scope

MetaEarth-MM addresses the scarcity of complete paired observations in multimodal remote sensing, where fully paired and well-aligned observations are rare and where traditional pairwise image-to-image translation scales poorly as the number of modalities increases (Yu et al., 19 May 2026). The model’s central premise is that all modalities are different observations of the same underlying scene. Rather than mapping appearance directly across modalities, it infers a latent scene variable from available inputs and conditions each target modality on this variable, thereby decoupling scene understanding from modality-specific generation (Yu et al., 19 May 2026).

The supported modalities are RGB, SAR, NIR, PAN, and OSM. These differ substantially in sensing physics and appearance: RGB captures visible spectral appearance, SAR captures microwave backscatter and includes speckle noise, NIR highlights vegetation and related spectral responses, PAN provides high-resolution grayscale optical imagery, and OSM consists of rendered map images encoding semantic layouts such as roads and buildings (Yu et al., 19 May 2026). EarthMM, the associated dataset, covers 0.5–10 m/pixel and includes global multi-resolution imagery, with RGB spanning all three major resolution bands, OSM and PAN concentrated at higher resolution, and SAR and NIR prominent in the medium range (Yu et al., 19 May 2026).

The framework is explicitly designed to unify three regimes within one model: paired joint generation, any-to-any translation, and completion of missing modalities (Yu et al., 19 May 2026). A plausible implication is that MetaEarth-MM is intended not only as a generator but also as a generalized cross-modal modeling substrate for Earth observation tasks in which source modality availability is variable.

2. Scene-centered joint modeling

The defining methodological claim of MetaEarth-MM is its scene-centered joint modeling paradigm. The model treats multimodal observations as conditionally independent given an underlying scene representation, formalized as

$$p(\mathbf{x}_i, \mathbf{x}_j | \mathbf{s}) = p(\mathbf{x}_i | \mathbf{s}) \, p(\mathbf{x}_j | \mathbf{s}).$$

This assumption motivates a design in which a shared scene representation drives independent modality-specific decoders (Yu et al., 19 May 2026). The stated purpose is to avoid the interference and optimization instability that can arise when heterogeneous modality distributions are entangled through direct appearance-level cross-modal mapping (Yu et al., 19 May 2026).

MetaEarth-MM adopts continuous flow matching with rectified flows to jointly learn marginal, conditional, and joint generation. For modality $m_i$, the latent trajectory and velocity are written as

$$\mathbf{z}_i^t = (1-t)\mathbf{z}_i^0 + t\mathbf{z}_i^1$$

and

$$\mathbf{v}_i = \frac{\mathrm{d}\mathbf{z}_i^t}{\mathrm{d}t} = \mathbf{z}_i^1 - \mathbf{z}_i^0.$$

The paper contrasts this with appearance-level conditional modeling, which predicts a target velocity given a clean source latent, and then generalizes to joint modeling in which both modalities are perturbed independently and dual velocities are predicted (Yu et al., 19 May 2026). Independent time sampling lets the same model learn unconditional, conditional, and joint regimes (Yu et al., 19 May 2026).

This formulation is important because paired joint generation requires modeling the joint distribution of modalities so that co-occurrence and spatial correspondence are preserved, while any-to-any translation requires conditioning on any available modality or subset to synthesize any target modality or set of targets, including directions not present in training pairs (Yu et al., 19 May 2026). This suggests that the scene variable functions as a latent coordination mechanism that subsumes many pairwise translators within a single parameterization.

3. Architecture and training objective

MetaEarth-MM uses a decoupled architecture trained end-to-end, consisting of a scene inference module $E_\theta$ and a modality-aware routed generator $G_\phi$ (Yu et al., 19 May 2026). The scene inference module is a DiT-based transformer that jointly processes noisy latents from available modalities to infer paired scene embeddings aligned to spatial positions. The modality-aware routed generator is a DiT-style conditional generator that predicts modality-specific velocity fields conditioned on the inferred scene tokens; self-attention is shared across modalities, while FFN branches are routed per modality to reduce interference (Yu et al., 19 May 2026).

In the scene inference module, noisy latents are patch-embedded with shared positional embedding and modality identifiers:

$$\mathbf{H}_k^{(0)} = \mathrm{Patch}_k(\mathbf{z}_k^{t_k}) + \mathbf{E}_{pos} + \mathbf{e}_{m_k}.$$

Tokens from both modalities are concatenated and processed by DiT blocks with shared Q/K/V projections so that self-attention runs over the full concatenated sequence and aggregates both intra-modal and cross-modal context (Yu et al., 19 May 2026). The module uses partition-adaptive LayerNorm, with per-modality modulation derived from timestep and modality embeddings:

$$[\boldsymbol{\gamma}_k^{(l)}, \boldsymbol{\beta}_k^{(l)}] = \mathrm{MLP}(\mathbf{c}_k)$$

and

$\bar{\mathbf{H}_k^{(l-1)}=\boldsymbol{\gamma}_k^{(l)} \odot \mathrm{LayerNorm}(\mathbf{H}_k^{(l-1)}) + \boldsymbol{\beta}_k^{(l)}.$

The scene inference output is written as

$[\hat{\mathbf{s}_i, \hat{\mathbf{s}_j] = \mathcal{E}_{\theta}(\mathbf{z}_i^{t_i}, \mathbf{z}_j^{t_j}, t_i, t_j, m_i, m_j).$

To regularize scene consistency, the model applies token-wise cosine similarity across corresponding spatial positions and a symmetric InfoNCE loss (Yu et al., 19 May 2026).

The modality-aware routed generator conditions on scene tokens and uses deterministic per-modality routing in FFN branches:

$m_i$0

It predicts velocities independently for each modality:

$m_i$1

The overall objective combines joint flow loss and scene consistency:

$m_i$2

with $m_i$3 (Yu et al., 19 May 2026).

Several training details are explicitly specified. Inputs are uniformly cropped or resampled to $m_i$4, positional embeddings are shared across modalities to preserve alignment, and EarthMM pairs are aligned by coordinates (Yu et al., 19 May 2026). A shared pre-trained VAE from latent diffusion is used for all modalities and remains frozen. The full model has approximately 600M parameters; the scene inference module uses 20 DiT blocks with hidden dimension 1024, the generator uses 4 DiT blocks with hidden dimension 1024, patch size is 2, and routed FFN branches are set to $m_i$5, one per modality (Yu et al., 19 May 2026). Training uses PyTorch, AdamW with learning rate $m_i$6, total batch size 256, 900k iterations, and NVIDIA A800 GPUs. Uniform pair sampling balances data, one modality is randomly dropped with probability 0.1 and replaced with pure noise to enable classifier-free guidance, and pseudo-PAN augmentation is introduced by grayscale conversion from RGB (Yu et al., 19 May 2026).

4. EarthMM dataset and supported generation tasks

EarthMM is the data substrate for MetaEarth-MM. It comprises 2.8 million multi-resolution global images with 2.2 million aligned pairs across the five modalities (Yu et al., 19 May 2026). The dataset integrates 15 public aligned datasets—OpenEarthMap-SAR, DFC23, DDHRNet, fMoW, MultiResSAR, OSDataset, QXS-SAROPT, SEN12MS, SpaceNet-3/-5/-6, SSL4EO-S12, WHU-OPT-SAR, and 3MOS—and also includes newly collected RGB–OSM pairs across 42 cities worldwide, including London, Beijing, New York, and Tokyo (Yu et al., 19 May 2026).

The RGB–OSM pairs are aligned by coordinates; repetitive homogeneous regions are removed; and preprocessing includes strict cleaning to remove severe noise, corrupted observations, and large cloud occlusions (Yu et al., 19 May 2026). The evaluation test sets are assembled from EarthMM pairs and include SAR-RGB (6826), NIR-RGB (6670), PAN-RGB (6670), and OSM-RGB (10,000) (Yu et al., 19 May 2026).

The framework supports several task families already defined in the paper. Paired joint generation includes directly sampling aligned pairs such as RGB-SAR, RGB-OSM, RGB-NIR, NIR-SAR, and PAN-SAR from noise (Yu et al., 19 May 2026). Any-to-any translation includes SAR→RGB, RGB→SAR, NIR→RGB, RGB→NIR, NIR→SAR, SAR→NIR, PAN→RGB, OSM→RGB, and RGB→OSM, while also supporting zero-shot directions not seen during training, such as OSM↔NIR or OSM↔SAR (Yu et al., 19 May 2026). The model also completes missing modalities by inferring the latent scene from partial observations and synthesizing the missing ones (Yu et al., 19 May 2026).

The inference process is described step by step: available modality images are encoded into latents using the shared VAE; independent timesteps are sampled; target latents are initialized from noise; scene inference produces aligned scene tokens; the generator predicts modality-specific velocities; the flow is integrated with an Euler–Maruyama sampler for 250 steps; and classifier-free guidance is applied with guidance scale 1.2 (Yu et al., 19 May 2026). No textual prompts are used; conditioning is provided by modality identifiers, timestep embeddings, and scene tokens (Yu et al., 19 May 2026).

5. Empirical performance and ablation findings

Evaluation uses FID for distributional realism, LPIPS for perceptual fidelity, SSIM for structural similarity, and for SAR additionally PSNR and FSIM because of speckle noise (Yu et al., 19 May 2026). Cross-modal consistency is measured by CAS, the Cross-modal Alignment Score, defined as

$m_i$7

Selected quantitative results reported in the paper include SAR→RGB with FID 38.11, LPIPS 0.4392, SSIM 0.2441, and CAS 59.82, outperforming Text2Earth with FID 49.43; RGB→SAR with FID 47.10, FSIM 0.6784, PSNR 13.34, and CAS 60.04; OSM→RGB with FID 43.04, LPIPS 0.5106, SSIM 0.2415, and CAS 70.04; RGB→OSM with FID 23.08, LPIPS 0.2101, SSIM 0.8411, and CAS 68.17; NIR→RGB with FID 28.45, LPIPS 0.4188, SSIM 0.3933, and CAS 67.59; and PAN→RGB with FID 30.58, LPIPS 0.3225, and SSIM 0.7622 (Yu et al., 19 May 2026).

For paired joint generation, OSM–RGB joint generation outperforms cascaded pipelines, with RGB FID 45.68 and CAS 64.45 for joint generation versus MetaEarth-MM Cascade FID 46.86 and CAS 62.42, while cascaded baselines such as Pix2Pix degrade to RGB FID 115.98 (Yu et al., 19 May 2026). Similar improvements are reported for SAR–RGB and NIR–RGB: SAR–RGB joint generation yields RGB FID 30.47 versus 34.85 for cascade and CAS 52.76 versus 50.90; NIR–RGB joint generation yields RGB FID 38.11 versus 39.92 and CAS 63.60 versus 61.57 (Yu et al., 19 May 2026).

The ablation results support the model’s structural claims. The decoupled architecture performs significantly better than a direct joint transformer, especially on large domain gaps such as SAR/RGB and OSM/RGB. Modality-routed FFN further reduces FID by avoiding cross-modality capacity competition. The scene consistency constraint improves CAS and convergence, clarifying the role of intermediate scene tokens. Training curves show faster convergence and lower final FID for the full MetaEarth-MM configuration (Yu et al., 19 May 2026).

The paper also reports zero-shot results for unseen modality pairs such as OSM–NIR and OSM–SAR, where outputs remain spatially coherent and semantically consistent (Yu et al., 19 May 2026). This suggests compositional generalization through the shared scene representation rather than memorization of specific training directions.

6. Downstream utility, lineage, and limitations

MetaEarth-MM is presented not only as a generative model but also as a model with downstream utility at both data and representation levels (Yu et al., 19 May 2026). In generative data augmentation for cross-modal matching, adding 10k generated pairs to 10k real pairs improves performance for LightGlue, LoFTR, and DKM on RGB–OSM and RGB–SAR benchmarks. Reported examples include LightGlue on RGB–OSM, where EPE improves from 3.757 to 3.494, PCK@1px from 0.0780 to 0.0959, and PCK@10px from 0.9647 to 0.9695; LoFTR on RGB–SAR, where EPE improves from 20.16 to 4.932, PCK@1px from 0.2102 to 0.3344, and PCK@10px from 0.7831 to 0.8828; and DKM on RGB–SAR, where EPE improves from 39.83 to 10.27 and PCK@10px from 0.1974 to 0.6647 (Yu et al., 19 May 2026).

For image-level domain adaptation on WHU-OPT-SAR, translating SAR or NIR into RGB before fine-tuning RGB-pretrained MaskFormer or DeepLabv3 improves segmentation. Reported values include MaskFormer on SAR, with mIoU 14.53→38.53 and mAcc 27.96→51.22, and on NIR, with mIoU 36.83→43.28 and mAcc 47.40→55.00; DeepLabv3 on SAR, with mIoU 28.49→33.58 and mAcc 37.46→43.45, and on NIR, with mIoU 35.17→39.04 and mAcc 46.99→50.92 (Yu et al., 19 May 2026). For zero-shot representation transfer on SEN12-MS, a lightweight classifier trained on scene tokens inferred by $m_i$8 transfers better across modalities than ResNet50 and DINOv3, with train RGB, test SAR accuracy 65.00% versus ResNet 18.57% and DINOv3 18.21%, train NIR, test SAR accuracy 65.00% versus ResNet 14.71% and DINOv3 20.57%, and average accuracy retention across modality transfer of 83.82% versus ResNet50 30.89% and DINOv3 38.03% (Yu et al., 19 May 2026).

Within the broader MetaEarth line, the 2024 "MetaEarth" model is a generative foundation model for global-scale optical remote sensing image generation based on a resolution-guided self-cascading diffusion framework, but it is explicitly optical-only and the paper states that it does not introduce an official multimodal variant called MetaEarth-MM (Yu et al., 2024). The 2026 MetaEarth-MM therefore represents a shift from multi-resolution optical generation toward unified multimodal generation centered on scene content (Yu et al., 19 May 2026, Yu et al., 2024).

Several limitations are explicitly stated. SAR speckle and extreme backscatter conditions can still challenge structural fidelity; performance varies with resolution and sensor specifics; training is fixed to $m_i$9 patches; only five modalities are currently supported; EarthMM integrates datasets with varying licenses and curation; and ethical considerations include the risk that synthetic imagery for environmental monitoring could misinform stakeholders, as well as privacy concerns in urban scenes and the need to disclose synthetic provenance and uncertainty (Yu et al., 19 May 2026). These constraints delimit the model’s current scope and indicate that scaling to additional sensors, native resolutions, and more rigorous governance remains an open problem.

A potential misconception is that "MetaEarth-MM" is a stable, unambiguous name across domains. The record is more complex. The 2026 remote-sensing preprint defines MetaEarth-MM as a unified multimodal generative foundation model (Yu et al., 19 May 2026). By contrast, the 2024 MetaEarth paper explicitly states that no official multimodal variant called MetaEarth-MM is introduced there (Yu et al., 2024), while the same label appears in synthesized descriptions tied to mm/sub-mm transient astronomy and to a conceptual reframing of the VLMSA proposal for Earth-analog detection (Koljonen et al., 16 Dec 2025, Saito et al., 2011). In current scholarly usage, the official and technically specified referent is the remote-sensing model in (Yu et al., 19 May 2026).