AnyMatch: Supercharging Universal Multi-Modal Image Matching with Large-Scale Single-View Images

Abstract: Multi-modal image matching is essential for visual localization and multi-sensor fusion, but it is hindered by the scarcity of large-scale training data with precise geometric annotations. Existing real-world datasets suffer from prohibitive costs, limited scene diversity, and errors in SfM-MVS pipelines, while synthetic methods struggle to maintain 3D geometric consistency or achieve photorealistic appearance. To address this, we propose AnyMatch, a novel framework that leverages abundant, easily accessible single-view images at minimal cost to generate rich multi-modal training data. AnyMatch integrates monocular depth estimation, 3D reprojection, diffusion-based inpainting, and crossmodal image translation to synthesize multi-view, multi-modal image pairs with 3D geometric fidelity. Crucially, our method provides annotations that strictly adhere to 3D geometric consistency through explicit 3D reprojection, avoiding SfM-MVS error accumulation. Furthermore, AnyMatch offers strong scalability, enabling controllable scene diversity and annotation difficulty via adjustable input and camera parameters. We construct Any-syn, a large-scale synthetic multi-modal dataset using AnyMatch. Experimental results show that matching networks (e.g., LoFTR, EDM, RoMa) fine-tuned on Any-syn achieve substantial performance gains on multi-modal benchmarks, exhibiting superior generalization and robustness compared to models trained on existing data.

• The paper presents a framework that synthesizes multi-view, multi-modal image pairs from single-view images with high 3D geometric fidelity.
• It employs novel view and modality transformations using depth estimation, differentiable rendering, and diffusion-based inpainting for robust matching.
• Empirical results show significant performance gains over baselines in cross-modal tasks, validating its scalable and economical design.

Authoritative Analysis of "AnyMatch: Supercharging Universal Multi-Modal Image Matching with Large-Scale Single-View Images" (2606.31077)

Motivation and Problem Formulation

Multi-modal image matching is a fundamental problem in computer vision, underpinning visual localization, cross-sensor fusion, and robust perception under varying environmental conditions. The principal bottleneck in advancing dense cross-modal matching is the scarcity of large-scale datasets with precise geometric ground truth, especially for pairs involving disparate sensing modalities (e.g., RGB-IR, RGB-Depth, RGB-Event). Traditional dataset construction approaches relying on SfM-MVS pipelines suffer from restrictive operational characteristics, error accumulation, and limited scene diversity, while synthetic workflows either fail to preserve geometric consistency or lack photorealism and physical fidelity. Consequently, the coverage and generalization of current data-driven matchers are strongly constrained by both the quality and diversity of available supervision.

AnyMatch Framework: Synthesis Pipeline and Innovations

AnyMatch introduces a generative framework for universal multi-modal matching, leveraging large repositories of single-view images to synthesize multi-modal, multi-view image pairs with high 3D geometric fidelity, at negligible acquisition cost. The core pipeline integrates:

• View Transformation: Single-view images are lifted to 3D via monocular depth estimation with Moge V1, perturbed for robustness, and reprojected under randomly sampled camera intrinsics and extrinsics. Differentiable rendering and diffusion-based inpainting (Stable Diffusion V2) fill occluded regions, yielding geometrically consistent novel views with accompanying depth and camera parameters.
• Modality Transformation: Cross-modal image translation is performed with modality-specific generative models (e.g., DiffV2IR for RGB-to-IR), monocular depth/normal estimation, and event stream simulation based on synthetic motion and brightness thresholds.
• Integration and Annotation: Pixel-level GT correspondences between views/modalities are established through explicit 3D reprojection, ensuring strict geometric consistency and avoiding cumulative errors typical of SfM-MVS.
• Sample-Level Geometric Consistency Verification (SGCV): High-fidelity supervision is further assured by filtering out pairs exhibiting hallucination or semantic inconsistency. Matching quality is verified with RoMa via endpoint error (EPE) and PCK thresholds, retaining only geometrically consistent pairs for training.

AnyMatch decouples data generation from hardware constraints, supports adjustable scene diversity and annotation difficulty, and generalizes to arbitrary modalities and environments by leveraging the statistical richness of single-view image repositories.

Empirical Evaluation: Quantitative and Qualitative Results

Experiments were conducted using LoFTR, EDM, and RoMa, with fine-tuning on the synthesized Any-syn dataset derived from GLDv2 and SA-1B. Datasets cover RGB-RGB/IR/Depth/Normal/Event modalities; real-world evaluation is performed on METU-VisTIR (RGB-IR), DIODE (RGB-Depth/Normal), DSEC (RGB-Event), and MMIM (medical and remote sensing).

Key Results:

• On Any-syn, models fine-tuned with AnyMatch consistently outperform baselines and MINIMA-fine-tuned models across all modalities. RoMa$_\text{AnyMatch}$ achieves $AUC@10^\circ$ of $69.88\%$ for RGB-IR, an absolute gain of $9.26\%$ over RoMa$_\text{MINIMA}$, and $21.23\%$ compared to vanilla RoMa.
• For highly challenging RGB-Event matching, EDM$_\text{AnyMatch}$ reaches $AUC@10^\circ$ of $9.32\%$ vs. $3.30\%$ (MINIMA), demonstrating superior robustness across modality gaps and low-information regimes.
• On real datasets, AnyMatch-trained models maintain competitive or superior performance. EDM$AUC@10^\circ$0 achieves $AUC@10^\circ$1 of $AUC@10^\circ$2 for RGB-IR, exceeding $AUC@10^\circ$3 of EDM$AUC@10^\circ$4; LoFTR$AUC@10^\circ$5 and RoMa$AUC@10^\circ$6 also establish high accuracy across seen and unseen modalities.
• Zero-shot evaluations on MMIM reveal strong generalization, with AnyMatch-trained models yielding distinct improvements for medical and remote sensing cross-modal tasks.

Ablation and Sensitivity Studies

Comprehensive ablation studies clarify the contributions of each pipeline component:

• Multi-modal joint training delivers substantial improvements in generalization over single-modality training.
• SGCV filtering at $AUC@10^\circ$7 maximizes model robustness, reducing hallucination artifacts.
• Fine-tuning pre-trained models is critical; training from scratch reduces AUC@20$AUC@10^\circ$8 by nearly $AUC@10^\circ$9.
• 3D view transformation-based synthesis outperforms 2D homography-based methods in matching accuracy, affirming the necessity of physically plausible disparity and occlusion modeling.
• Analyses with varying camera intrinsic/extrinsic parameters confirm both the practical utility and resilience of randomized sampling within chosen intervals.

Implications and Future Directions

Practically, AnyMatch establishes a scalable, economical path to high-quality supervision for universal cross-modal matching, bypassing the prohibitive costs and coverage limits of real multi-sensor acquisition. Theoretically, it provides a mechanism to enforce geometric constraints and minimize annotation error, thereby strengthening the capacity for learning modality-invariant features and achieving robust matching in long-tail, real-world scenarios. The methodology also supports curriculum learning and controllable dataset synthesis, enabling tailored benchmarking and training protocols.

Potential extensions include broader integration of advanced generative networks for modality translation, employing domain adaptation to further mitigate residual gaps between synthetic and real data, and exploration of unsupervised learning protocols leveraging AnyMatch-derived pseudo-supervision. The SGCV paradigm may be adapted for self-supervised training regimes or active selection in lifelong adaptation settings.

Conclusion

AnyMatch advances the state-of-the-art in universal multi-modal image matching, introducing a geometry-supervised synthesis pipeline that converts abundant single-view images into multi-modal, multi-view pairs with pixel-level 3D fidelity. The approach successfully fine-tunes and augments leading matchers (LoFTR, EDM, RoMa), conferring strong performance gains across synthetic and real-world evaluation benchmarks and enabling generalization to previously unseen modalities. The framework provides a robust, scalable solution to data scarcity and geometric annotation challenges, with broad implications for the development and deployment of cross-modal perception systems in open environments.