MetaEarth-MM: Unified Multimodal Remote Sensing Image Generation with Scene-centered Joint Modeling

Abstract: Multi-modal remote sensing images are vital for Earth observation, yet complete paired observations are often scarce in practice. Existing generative methods commonly address this problem through isolated pairwise modality translation, but their versatility and scalability remain limited as the number of modalities and generation tasks increases. Here, we develop a generative foundation model MetaEarth-MM for multi-modal remote sensing imagery, enabling paired joint generation and any-to-any translation across five modalities within a unified model. Recognizing the intrinsic scene consistency underlying multi-modal observations, we introduce a scene-centered joint modeling paradigm in MetaEarth-MM. Unlike previous methods that rely on direct appearance-level cross-modal mapping, our model organizes the generation around the underlying scene content. Specifically, MetaEarth-MM adopts a decoupled architecture that first infers a latent scene representation from available observations, and then generates target modalities conditioned on this intermediate state. To support training, we further construct EarthMM, a large-scale dataset comprising 2.8 million multi-resolution global images with 2.2 million aligned pairs. Extensive experiments demonstrate that MetaEarth-MM not only exhibits strong generative capability and robust generalization across diverse generation tasks, but also supports downstream tasks at both data and representation levels, highlighting its potential as a general foundation model for cross-modal Earth observation. The code and dataset will be available at https://github.com/YZPioneer/MetaEarth-MM.

• The paper introduces a unified, scene-centered framework that decouples shared scene representation from modality-specific synthesis.
• It leverages a continuous flow matching strategy, demonstrating superior performance in both paired and unconditional multimodal generation.
• Empirical results show enhanced zero-shot translation and improved downstream tasks via robust cross-modal semantic alignment.

MetaEarth-MM: Scene-Centered Unified Multimodal Remote Sensing Image Generation

Introduction

MetaEarth-MM establishes a new paradigm for generative modeling in Earth observation by moving from isolated, pairwise cross-modal translation to unified, scene-centered joint modeling of multi-modal remote sensing imagery. The approach targets the intrinsic scene-level consistency present across diverse sensing modalities—including RGB, SAR, NIR, PAN, and OSM—and decouples shared scene representation learning from modality-specific synthesis. This enables the model to support both unconditional paired joint generation and any-to-any translation within a single scalable framework, addressing limitations of modality-centric translation techniques.

Figure 1: Comparison between previous pairwise modality translation (left) and MetaEarth-MM's scene-centered joint modeling (right); the right approach anchors generation on a learned latent scene representation.

From Pairwise Translation to Scene-Centered Joint Modeling

Prior works in multi-modal remote sensing focus on dedicated pairwise translation networks, which cannot scale with both the number of modalities and the exponential growth in possible translation directions. MetaEarth-MM formulates the joint modeling of multi-modal image distributions using the continuous flow matching framework. Instead of direct appearance mapping, the model infers a shared latent scene embedding from noisy paired observations, then synthesizes target modalities conditioned on this scene representation. This strategy ensures that the generative process is robust to missing or unpaired modalities, and directly enables paired and unconditional generation.

The theoretical foundation is a generative decoupling: the multi-modal joint distribution $p(\mathbf{x}_i, \mathbf{x}_j|s)$ factors into independent conditional distributions on the scene latent $s$. This decoupling, instantiated in flow-matching generative models, solves the cross-modal interference and optimization issues that arise in direct joint modeling.

Architecture and Algorithmic Design

The core architecture consists of two decoupled modules (Figure 2):

• Scene Inference Module: Processes concatenated noisy latents from multiple modalities with a DiT backbone, yielding modality-specific scene tokens. Cross-modal aggregation is realized via shared-attention transformers, while partition-adaptive normalization and dedicated patch embeddings account for modality and noise-level heterogeneity.
• Modality-Aware Routed Generator: Synthesizes each modality’s latent velocity field in the flow-matching process, conditioned on the corresponding scene embedding and facilitated by deterministic FFN routing. This provides distinct parameter branches for each modality, alleviating cross-modality modeling capacity contention.

Scene-level consistency is enforced with a symmetric InfoNCE regularization across scene tokens inferred from paired modalities.

Figure 2: MetaEarth-MM architecture: a decoupled pipeline with explicit scene inference and modality-specific generation, regularized for scene consistency.

Large-Scale Data Foundation: EarthMM

Modeling at global scale with strong generalization requires adequately diverse, aligned multi-modal data. The constructed EarthMM dataset aggregates 2.8M images (with 2.2M aligned pairs) across five modalities, covering broad spatial resolutions (0.5–10 m/pixel) and global geography. This surpasses prior resources in both scale and modality variety, providing dense supervision for learning robust scene-modality correspondence.

Figure 3: Overview of EarthMM: spatial coverage, modality pair statistics, and spatial resolution distribution.

Empirical Results: Generative Performance and Robustness

Paired Joint Generation and Cross-Modal Translation

MetaEarth-MM achieves strong results in both unconditional (from noise) and conditional settings. Unconditional joint generation produces structurally consistent, modality-respectful image pairs. Cross-modal translation is evaluated on SAR/RGB, OSM/RGB, NIR/RGB, PAN/RGB, and their inverse mappings, consistently outperforming specialized baselines (Pix2Pix, Palette, BiBBDM, Text2Earth, BBDM) across FID, LPIPS, SSIM, and CAS metrics.

Figure 4: Qualitative results of MetaEarth-MM in unconditional paired generation, cross-modal translation, and zero-shot unseen tasks.

Figure 5: MetaEarth-MM exceeds translation quality and cross-modal consistency vs. leading diffusion and GAN methods.

Zero-Shot Generalization

A central claim, zero-shot translation for modality pairs unseen during training (e.g., OSM–NIR, OSM–SAR), is validated: the model composes previously learned scene-modality relationships for semantically and structurally plausible outputs without direct supervision. This is enabled by the explicit scene-centered design and is a marked theoretical and practical distinction from prior approaches.

Convergence and Ablations

Systematic ablation reveals:

• The decoupled scene-and-generator structure yields significant FID and CAS improvements, especially for high-gap modality pairs.
• Modality-routed FFN reduces parameter contention, further improving generative fidelity.
• Scene consistency constraint maximizes cross-modal semantic alignment.

MetaEarth-MM demonstrates superior optimization stability and convergence speed.

Figure 6: FID training curves: full MetaEarth-MM converges faster and to better minima than ablated alternatives.

Downstream Utility and Representation Transfer

Beyond generative metrics, MetaEarth-MM broadly benefits Earth observation downstream:

• Generative Data Augmentation: Synthetic paired data from MetaEarth-MM substantially improves cross-modal image matching tasks (LightGlue, LoFTR, DKM), particularly in high-gap settings (e.g., RGB–SAR).
• Data-centric Domain Adaptation: Translating non-RGB modalities into the RGB domain increases semantic segmentation performance using RGB-pretrained MaskFormer and DeepLabv3, showing that translated images retain class semantics and leverage existing robust visual priors.
• Zero-Shot Representation Transfer: Without retraining, the learned scene representations allow single-domain classifiers to generalize with high accuracy to other modalities, greatly surpassing convnet and ViT baselines.

Implications and Future Developments

The work establishes the value of explicit scene-centered joint modeling for scalable, robust multi-modal generation, supporting fine-grained control, translation, and augmentation on a unified latent basis. Practically, MetaEarth-MM can serve as a foundation for downstream Earth observation systems, enabling data synthesis and cross-modal interpretation even in regimes of incomplete pairing and rare modalities.

Theoretically, decoupled scene-modality architectures and flow-based training regimes address fundamental limitations in cross-modal distribution modeling previously encountered in both GAN- and diffusion-based paradigms. The model’s compositionality and modularity suggest compatibility with compositional and few-shot conditioning for broader classes of spatial-sensor data, and scalability to further modalities (e.g., hyperspectral, LiDAR) and higher spatial-temporal resolutions.

Conclusion

MetaEarth-MM introduces a unified, scalable framework for generative modeling of multi-modal remote sensing imagery. Scene-centered joint modeling, realized via explicit decoupling and optimized with large-scale, well-aligned data, advances the state-of-the-art in both conditional and unconditional cross-modal image synthesis, zero-shot transfer, and downstream utility. The results indicate future directions toward universal, modality-agnostic foundation models for geospatial AI and Earth observation.