DistillMatch: Cross-Modal Image Matching

• DistillMatch is a multimodal image matching framework that achieves pixel-level correspondence by distilling semantic features from Vision Foundation Models.
• It combines teacher-student knowledge transfer, explicit modality category injection, and GAN-driven data augmentation to overcome modality gaps and data scarcity.
• Empirical evaluations show superior performance in tasks like relative pose and homography estimation, demonstrating its robustness on challenging cross-modal benchmarks.

DistillMatch is a multimodal image matching framework that achieves pixel-level feature correspondence between images of different modalities—such as visible and infrared—by leveraging knowledge distillation from large-scale Vision Foundation Models (VFMs). The method combines high-level semantic feature transfer from powerful teacher models (e.g., DINOv2, DINOv3) with lightweight student networks, explicit preservation and injection of modality category information, and GAN-based data augmentation to address the challenges of modality gap and data scarcity in cross-modal image matching (Yang et al., 19 Sep 2025).

1. Multimodal Image Matching with Vision Foundation Model Distillation

DistillMatch addresses the core difficulty in multimodal image matching: extracting robust, modality-agnostic features for accurate, pixel-wise correspondences when input images differ dramatically in appearance (e.g., visible-light vs. infrared). Conventional deep learning approaches relying on extracting common or invariant features often perform poorly due to limited annotated data and insufficient adaptability to diverse scenarios.

DistillMatch leverages pre-trained VFMs (specifically DINOv2 ViT-S/14 and DINOv3 ViT-L/16) as teacher networks. These models, trained on diverse, large-scale image corpora, provide high-level semantic representations that generalize well across modalities. Through knowledge distillation, these representations are transferred to a lightweight student network tailored for computational efficiency and direct supervision from image matching labels, facilitating robust, modality-independent feature extraction.

2. Knowledge Distillation Process and Loss Formulation

Knowledge distillation in DistillMatch employs a teacher–student paradigm. The objective is for the lightweight student vision transformer to replicate the semantic features produced by the VFM teacher on corresponding input images, with additional adaptation to the specifics of the matching task.

The total distillation loss comprises three key terms:

• Mean Squared Error (MSE) Loss: Enforces direct pixel-level similarity between normalized teacher and student feature maps.
• Gram Matrix Loss: Preserves spatial relationships by aligning the pairwise similarity structure of features, capturing co-activation patterns across image regions.
• Kullback–Leibler (KL) Divergence Loss: Aligns probabilistic feature distribution profiles output by student and teacher networks.

These are combined as: $$L_\mathrm{KD} = \alpha \cdot L_\mathrm{MSE} + \beta \cdot L_\mathrm{Gram} + \gamma \cdot L_\mathrm{KL}$$ where $\alpha$, $\beta$, $\gamma$ are weighting coefficients. This tri-term loss drives the student to both inherit semantic knowledge and adapt it to the matching task for cross-modal consistency.

3. Feature Extraction, Modality Category Injection, and Category-Enhanced Guidance

DistillMatch employs parallel feature extraction streams:

• VFM Pathway: High-level semantic features, robust to cross-modality variation, are distilled from the VFM teacher.
• Multiscale ResNet Branch: Local texture features at multiple scales are computed to retain fine-grained geometric information.

A Category-Enhanced Feature Guidance Module (CEFG) is introduced to address the residual modality gap. This module maintains:

• Learnable modality category embeddings for each input type (visible, infrared, etc.).
• Concatenation of the category-specific embedding with low-level features, followed by transformer processing.
• Element-wise injection of these guided embeddings into the deep features of the opposite modality branch.

This process selectively integrates modality-dependent contextual cues—enabling the subsequent matching module to reliably pair both shared and unique features across modalities.

4. Synthetic Data Augmentation with V2I-GAN

To counteract the scarcity of labeled multimodal training data, DistillMatch introduces V2I-GAN—a visible-to-pseudo-infrared generative adversarial network inspired by CycleGAN and PearlGAN. The architecture consists of:

• Two generators $(G_{VI}, G_{IV})$ for bi-directional translation between visible and infrared domains.
• Two discriminators $(D_V, D_I)$ to ensure synthetic images resemble each modality's real distribution.
• Encoder-decoder generators that incorporate a Semantic-Texture Fusion and Aggregation (STFA) module to ensure semantic preservation during translation.

By generating synthetic pseudo-infrared images from visible inputs (and vice versa), more annotated data can be created without manual intervention. Scene labels and geometric structure are preserved, and the overall diversity and quality of the training set improve, resulting in better model generalization, especially in low-resource or zero-shot settings.

5. Experimental Validation and Performance

DistillMatch demonstrates strong empirical performance across challenging multimodal image matching benchmarks:

• Relative Pose Estimation (METU-VisTIR dataset): Achieves higher area under curve (AUC) values than prior state-of-the-art methods under both cloudy–cloudy and cloudy–sunny imaging conditions.
• Homography Estimation: Yields superior reprojection error metrics in UAV remote sensing, indoor, night, and haze scenarios, indicating precise cross-modal geometric alignment.
• Zero-shot Generalization: When evaluated on previously unseen modality pairs (optical-SAR, optical-map, etc.), DistillMatch maintains high accuracy, underscoring the robustness of the distilled semantic features and augmentation strategy.

Quantitative and qualitative results indicate that combining VFM-based knowledge distillation, selective injection of modality category information, and V2I-GAN–augmented training data leads to superior pixel-level correspondence accuracy compared to competing algorithms.

6. Mathematical Formulations

DistillMatch relies on several key loss functions:

Loss Term	Mathematical Expression	Purpose
MSE Loss	$$L_\mathrm{MSE} = \frac{1}{N} \sum_{i=1}^N \| \frac{F_\mathrm{tea}}{\|F_\mathrm{tea}\|_2} - \frac{F_\mathrm{stu}}{\|F_\mathrm{stu}\|_2} \|^2$$	Pixel-level feature alignment
Gram Matrix Loss	$$L_\mathrm{Gram} = \frac{1}{N} \sum_{i=1}^N \| G(F_\mathrm{tea}) - G(F_\mathrm{stu}) \|_2^2$$, $G(F) = \frac{F F^T}{HW}$	Spatial correlation structure preservation
KL Divergence Loss	$$L_\mathrm{KL} = D_\mathrm{KL}(F_\mathrm{tea} \| F_\mathrm{stu})$$	Distributional feature alignment
Category Loss (CEFG)	$$L_\mathrm{ce} = CE(P_\mathrm{vis}, [0,1]) + CE(P_\mathrm{ir}, [1,0])$$	Explicit modality category guidance

Additional loss components for coarse-level focus loss and subpixel refinement are also employed, precise in their mathematical definitions to enforce geometric and probabilistic matching objectives.

7. Impact, Applications, and Future Directions

DistillMatch establishes a new multimodal image matching paradigm by integrating three orthogonal advances:

• Cross-modal semantic knowledge transfer via VFM distillation.
• Category-enhanced feature guidance preserving both shared and unique modality cues.
• Generative augmentation for robust generalization in low-label regimes.

This comprehensive approach improves the accuracy of critical applications in remote sensing, medical imaging, and autonomous vision under modality gaps and scarce annotation. Suggested future work includes application to a broader array of modalities (e.g., SAR, depth), leveraging other large-scale pretraining paradigms, and refined student network optimization for deployment on resource-constrained devices. There is also an explicit proposal to further explore pretraining coverage and the efficiency of the category injection mechanism for even greater cross-modal adaptability.

In summary, DistillMatch represents a systematic solution to multimodal matching via VFM-based knowledge transfer, category-guided feature injection, and data augmentation, validated empirically to outperform existing alternatives on several public benchmarks (Yang et al., 19 Sep 2025).