MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching (2501.11299v3)

Abstract: Many keypoint detection and description methods have been proposed for image matching or registration. While these methods demonstrate promising performance for single-modality image matching, they often struggle with multimodal data because the descriptors trained on single-modality data tend to lack robustness against the non-linear variations present in multimodal data. Extending such methods to multimodal image matching often requires well-aligned multimodal data to learn modality-invariant descriptors. However, acquiring such data is often costly and impractical in many real-world scenarios. To address this challenge, we propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching using only single-modality training data. Specifically, we propose a novel latent feature aggregation module and a cumulative hybrid aggregation module to enhance the base keypoint descriptors trained on single-modality data by leveraging pre-trained features from Stable Diffusion models. %, our approach generates robust and invariant features across diverse and unknown modalities. We validate our method with recent keypoint detection and description methods in three multimodal retinal image datasets (CF-FA, CF-OCT, EMA-OCTA) and two remote sensing datasets (Optical-SAR and Optical-NIR). Extensive experiments demonstrate that the proposed MIFNet is able to learn modality-invariant feature for multimodal image matching without accessing the targeted modality and has good zero-shot generalization ability. The code will be released at https://github.com/lyp-deeplearning/MIFNet.

• The paper presents MIFNet, a network utilizing diffusion models to learn modality-invariant features for robust, generalizable multimodal image matching.
• MIFNet achieves zero-shot generalization and high accuracy in multimodal image matching across diverse datasets, including retinal and remote sensing, without requiring paired training data from target modalities.
• The network significantly improves registration success rates on challenging datasets like CF-FA retinal images (e.g., 64.1% SRR with SuperPoint) and outperforms state-of-the-art methods in remote sensing.

MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching

The paper presents MIFNet, a network specifically devised to address the challenges associated with multimodal image matching. This task is notoriously complex due to inherent differences in geometric and intensity transformations present across different imaging modalities. The traditional approaches, although effective for single-modality scenarios, frequently falter in multimodal contexts due to their inability to produce robust, invariant descriptors which can span diverse modalities. In this landscape, MIFNet emerges as a promising methodology, leveraging modality-invariant features derived from diffusion models, specifically tailored to maintain robust keypoint descriptors in multimodal image environments.

Key innovations in MIFNet's architecture include the utilization of Stable Diffusion models to harness pre-trained features which are subsequently refined through two advanced modules: the Latent Feature Aggregation (LFA) and Cumulative Hybrid Aggregation (CHA) modules. The LFA module refines coarse features extracted from diffusion models with a Gaussian mixture model to enhance semantic and modality invariance. These refined features interact within the CHA module, which employs multi-layer attention mechanisms to integrate base features obtained from traditional single-modality detectors and the refined semantic features to produce a robust, modality-invariant feature set.

The ability of MIFNet to accomplish high-accuracy multimodal image matching without the need for paired training data from the target modalities is a distinguishing feature of this approach. This eliminates the reliance on costly and logistically challenging acquisition of well-aligned cross-modal data, a requirement of many other methods. The zero-shot generalization capability of MIFNet is particularly impressive, achieving improved alignment accuracy across both multimodal retinal and remote sensing images.

In the field of medical imaging, MIFNet was evaluated on three retinal datasets (CF-FA, CF-OCT, EMA-OCTA), demonstrating substantial improvements over state-of-the-art techniques in success registration rates (SRR), with significant boosts from baseline methods. For instance, when integrated with SuperPoint, a 64.1% SRR was achieved on the CF-FA dataset, greatly surpassing previous results that struggled with the non-linear intensity discrepancies and intricate geometric distortions characteristic of those datasets.

Similarly, MIFNet showed remarkable performance improvements in remote sensing applications. Tested against Optical-SAR and Optical-NIR datasets, it consistently outperformed not only handcrafted algorithms like OS-SIFT and RIFT2 but also newer multimodal learning-based methods that demanded modality-specific retraining or alignment pre-processing.

Theoretical implications of this work stretch into the domains of cross-domain feature learning and self-supervised learning, particularly within the context of utilizing diffusion models for tasks beyond mere image generation. Practically, MIFNet offers a viable solution for applications requiring efficient and accurate image matching across varied modalities like medical imaging diagnostics, remote sensing data interpretation, and autonomous navigation systems.

Future research trajectories may explore refining the computational efficiency of MIFNet’s feature extraction and aggregation processes, potentially through more streamlined architectural designs or novel attention mechanisms. Furthermore, expanding the scope of evaluation to include other modalities or integrating with transformers could yield insights into the adaptability and robustness of diffusion-based feature extraction in broader contexts.

Ultimately, MIFNet stands as a compelling advance in tackling the complex challenges associated with multimodal image matching, offering both substantial performance enhancements and significant reductions in training data constraints.