PromptMID: Modal Invariant Descriptors Based on Diffusion and Vision Foundation Models for Optical-SAR Image Matching (2502.18104v1)

Abstract: The ideal goal of image matching is to achieve stable and efficient performance in unseen domains. However, many existing learning-based optical-SAR image matching methods, despite their effectiveness in specific scenarios, exhibit limited generalization and struggle to adapt to practical applications. Repeatedly training or fine-tuning matching models to address domain differences is not only not elegant enough but also introduces additional computational overhead and data production costs. In recent years, general foundation models have shown great potential for enhancing generalization. However, the disparity in visual domains between natural and remote sensing images poses challenges for their direct application. Therefore, effectively leveraging foundation models to improve the generalization of optical-SAR image matching remains challenge. To address the above challenges, we propose PromptMID, a novel approach that constructs modality-invariant descriptors using text prompts based on land use classification as priors information for optical and SAR image matching. PromptMID extracts multi-scale modality-invariant features by leveraging pre-trained diffusion models and visual foundation models (VFMs), while specially designed feature aggregation modules effectively fuse features across different granularities. Extensive experiments on optical-SAR image datasets from four diverse regions demonstrate that PromptMID outperforms state-of-the-art matching methods, achieving superior results in both seen and unseen domains and exhibiting strong cross-domain generalization capabilities. The source code will be made publicly available https://github.com/HanNieWHU/PromptMID.

• The paper introduces PromptMID, a novel framework using diffusion and vision foundation models guided by text prompts to create robust modal-invariant descriptors for optical-SAR image matching.
• PromptMID utilizes features from large pretrained models and text prompts based on land use classification to achieve strong cross-domain generalization and overcome geometric and radiometric differences between modalities.
• Experimental validation shows PromptMID achieves 100% matching success on challenging multi-modal datasets like WHU-OPT-SAR and demonstrates superior generalization ability on diverse unseen datasets.

Modal Invariant Descriptors for Optical-SAR Image Matching

This paper, titled "PromptMID: Modal Invariant Descriptors Based on Diffusion and Vision Foundation Models for Optical-SAR Image Matching," addresses the crucial challenge of image matching between optical and Synthetic Aperture Radar (SAR) imagery. The complexity and variability inherent in multi-modal image datasets, particularly those arising from geometric and radiometric differences, necessitate robust solutions for image matching tasks in remote sensing applications.

Methodology Overview

The core innovation of this work is the development of PromptMID, a framework that constructs modality-invariant descriptors using text prompts grounded in land use classification. This is augmented by exploiting pre-trained diffusion models and visual foundation models (VFMs) to enhance cross-domain generalization and robustness.

Key Components:

• Diffusion Models and VFMs: The authors utilize diffusion models and VFMs to extract multi-scale features that remain consistent across variable scenes, thereby facilitating robust cross-domain matching. These models, pretrained on vast datasets, confer strong generalization capabilities that are critical in remote sensing.
• Text Prompts: Text prompts based on land use classification provide semantic guidance. This is a novel approach for optical-SAR matching, using semantic information as priors to guide feature extraction, enhancing the identification of invariant features across modalities.
• Feature Aggregation: A multi-scale aware aggregation (MSAA) module fuses features extracted at various granularities, consolidating both coarse and fine details from optical and SAR images. This module dynamically adjusts feature importance through learnable mechanisms that balance representation detail and invariance.

Experimental Validation

The empirical evaluation is extensive, utilizing datasets from varied geographical regions to test both seen and unseen image datasets. On the WHU-OPT-SAR dataset, a challenging multi-modal imagery dataset, PromptMID achieved a 100% success rate in matching performance, representing an improvement over both traditional handcrafted methods and other state-of-the-art machine learning approaches.

Generalization Tests:

• The tests on unseen datasets (SEN1-2, A OPT-SAR, B OPT-SAR) demonstrate the superior generalization ability of PromptMID compared to current methodologies. The SR (Success Rate) achieved is markedly higher across these datasets, with a notable reduction in RMSE (Root Mean Square Error), underscoring the method's robustness in diverse and challenging scenarios.

Implications and Future Directions

The successful integration of textual semantic priors with foundational model features suggests a promising direction for future remote sensing applications, where domain adaptation is crucial. This work opens avenues for utilizing large-scale pretrained models to bridge the optical-SAR domain gap more effectively, allowing for reduced reliance on domain-specific fine-tuning.

Future advancements could potentially explore:

1. Enhanced incorporation of other foundation models tailored specifically for geospatial data.
2. Refining the semantic guidance mechanism with additional geographic data sources.
3. Improving computational efficiency to facilitate real-time applications and processing of larger datasets.

In conclusion, the introduction of PromptMID sets a precedent for future research in optical-SAR image matching, providing a robust framework to overcome longstanding challenges in domain generalization and feature consistency across diverse sensing modalities. This paper offers significant insights and tools for the ongoing advancement of image matching technologies in the field of remote sensing.