AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities (2412.14123v3)

Abstract: Geospatial models must adapt to the diversity of Earth observation data in terms of resolutions, scales, and modalities. However, existing approaches expect fixed input configurations, which limits their practical applicability. We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and scale-adaptive spatial encoders, allowing us to train a single model on highly heterogeneous data in a self-supervised manner. To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of 5 multimodal datasets with varying characteristics and $11$ distinct sensors. We then train a single powerful model on these diverse datasets simultaneously. Once fine-tuned or probed, we reach state-of-the-art results on the test sets of GeoPlex and for 6 external datasets across various environment monitoring tasks: land cover mapping, tree species identification, crop type classification, change detection, climate type classification, and segmentation of flood, burn scar, and deforestation. The code and models are available at https://github.com/gastruc/AnySat.

• The paper presents AnySat, an EO model that processes diverse resolutions, scales, and modalities using a novel JEPA and scale-adaptive patch encoding.
• It employs a self-supervised learning framework combining contrastive and predictive losses to robustly capture multimodal features.
• The model achieves state-of-the-art results in applications like tree species identification and land cover mapping, demonstrating strong adaptability.

Overview of AnySat: An Earth Observation Model for Diverse Resolutions, Scales, and Modalities

The paper introduces AnySat, a model designed to address the diverse and heterogeneous nature of Earth Observation (EO) data. The model can handle various resolutions, scales, and modalities, which challenge existing EO models due to their expectation of fixed input configurations. AnySat leverages a novel Joint Embedding Predictive Architecture (JEPA) alongside scale-adaptive spatial encoders to unify the training on heterogeneous datasets in a self-supervised manner.

Methodological Contributions

1. Scale-Adaptive Patch Encoding: AnySat integrates a scale-adaptive patch encoder, which allows for the handling of different spatial resolutions seamlessly. This encoder processes patches of varying sizes without the need for rescaling the input data, maintaining size consistency in embedded vectors while accommodating changes in patch size.
2. Joint Embedding Predictive Architecture (JEPA): The JEPA forms the backbone of the model, allowing it to predict missing data across modalities. This architecture forgoes the need for complex decoders by leveraging learned feature space reconstructions, thus simplifying the process and enhancing flexibility to handle various modalities.
3. Self-Supervised Learning Paradigm: The training methodology incorporates both contrastive and predictive losses. Modality and temporal masking ensure that the model learns robust features, unaffected by the absence of certain data channels or time points. The unique approach of combining a student-teacher model framework further reinforces the learning of consistent cross-modal representations.

Evaluation and Results

AnySat was evaluated using GeoPlex, a curated collection of multimodal EO datasets with a combined extent spanning five continents and 171 billion pixels. The datasets present significant variations in channels, revisit times, spatial resolutions, and geographical extents.

• Performance Benchmarks:

AnySat achieves state-of-the-art performance in tree species identification and land cover mapping, among other tasks, setting new benchmarks in classifications with improvements in weighted F1 and mIoU scores across different datasets. For instance, AnySat outperformed previous models on TreeSatAI-TS and PASTIS-HD datasets by significant margins.

• Generalization to New Datasets:

Importantly, AnySat can be fine-tuned or probed on new datasets, demonstrating strong adaptability even when the target datasets possess unique sensor configurations not present in the training data. This was evidenced by high performance on external datasets like SICKLE and BraDD-S1TS.

Practical Implications and Future Directions

The development of AnySat underscores a significant evolution in remote sensing and EO model design. It highlights the ability to embed multimodal, multi-resolution data into a single coherent model, facilitating application in various environmental monitoring tasks. The incorporation of JEPA provides a roadmap for future multimodal framework developments, enhancing self-supervised learning capabilities. Future research may explore integrating more diverse datasets for training, further optimizing the model's fine-tuning capabilities.

Conclusion

AnySat represents a versatile approach to solving the challenges posed by the variable nature of EO data. By leveraging self-supervised techniques and feature-predictive architectures, it sets a precedent for future EO models aiming to integrate and analyze data from a broad spectrum of resolutions and modalities. Its adaptability and state-of-the-art performance make it not just a powerful tool for current applications but also a foundational architecture for future EO applications and research endeavors.