CRFT: Consistent-Recurrent Feature Flow Transformer for Cross-Modal Image Registration

Abstract: We present Consistent-Recurrent Feature Flow Transformer (CRFT), a unified coarse-to-fine framework based on feature flow learning for robust cross-modal image registration. CRFT learns a modality-independent feature flow representation within a transformer-based architecture that jointly performs feature alignment and flow estimation. The coarse stage establishes global correspondences through multi-scale feature correlation, while the fine stage refines local details via hierarchical feature fusion and adaptive spatial reasoning. To enhance geometric adaptability, an iterative discrepancy-guided attention mechanism with a Spatial Geometric Transform (SGT) recurrently refines the flow field, progressively capturing subtle spatial inconsistencies and enforcing feature-level consistency. This design enables accurate alignment under large affine and scale variations while maintaining structural coherence across modalities. Extensive experiments on diverse cross-modal datasets demonstrate that CRFT consistently outperforms state-of-the-art registration methods in both accuracy and robustness. Beyond registration, CRFT provides a generalizable paradigm for multimodal spatial correspondence, offering broad applicability to remote sensing, autonomous navigation, and medical imaging. Code and datasets are publicly available at https://github.com/NEU-Liuxuecong/CRFT.

• The paper presents a transformer-based framework that reconceptualizes cross-modal registration as modality-independent feature flow estimation.
• The method integrates hierarchical multi-scale feature extraction, iterative discrepancy-guided optimization, and explicit spatial geometric transforms for robust sub-pixel alignment.
• Experimental results demonstrate state-of-the-art performance on OSdataset and RoadScene, with up to 42.5% improvement and reliable global and local precision.

Consistent-Recurrent Feature Flow Transformer (CRFT) for Robust Cross-Modal Image Registration

Motivation and Problem Setting

Cross-modal image registration remains a core challenge in computer vision due to large nonlinear appearance discrepancies and geometric variations across heterogeneous sensing modalities (e.g., optical, infrared, SAR). Traditional feature-based and photometric approaches often collapse in the presence of spectral gaps or strong affine transformations. Existing deep and transformer-based methods largely retain architectural biases toward homogeneous domains, leading to sub-optimal correspondence when modality gaps are dominant or severe deformations exist. To address these deficiencies, the paper introduces the Consistent-Recurrent Feature Flow Transformer (CRFT), which reconceptualizes registration as modality-independent feature flow estimation and introduces a unifying transformer-based, coarse-to-fine registration framework explicitly designed for multimodal robustness.

Architecture and Methodological Design

CRFT integrates several critical algorithmic advances into a single, coherent pipeline: hierarchical coarse-to-fine matching, iterative discrepancy-guided flow refinement, and an explicit spatial geometric transform (SGT) for recurrent sub-pixel alignment. The approach comprises the following staged components:

1. Hierarchical Multi-Scale Feature Extraction and Coarse Flow Estimation:

A shared CNN-based backbone extracts hierarchical features at multiple resolutions ($1/2$, $1/4$, $1/8$), ensuring that both coarse and fine geometric cues are accessible and mitigating modality-dependent appearance drift. The initial coarse flow field is estimated using transformer-based self-attention (SA) and cross-attention (CA) blocks applied to low-resolution features, with matching probabilities computed through a global correlation matrix and softmax scoring to provide stable, modality-agnostic correspondences.

2. Hierarchical Fine-Scale Feature Fusion and Local Flow Refinement:

For high-spatial-precision alignment, local windows are aggregated at each scale and fused through successive SA-CA modules, injecting progressively higher resolution contextual detail. This resolves local ambiguities and preserves structural consistency under strong appearance variations.

3. Iterative Discrepancy-Guided Flow Optimization (DGFO) with Spatial Geometric Transform:

The refinement core leverages a recurrent architecture coupling a fine-scale feature projection (FSFT), window-level SGT-based warping, and discrepancy estimation. At each iteration, local feature discrepancies between aligned windows are used to modulate attention weights, drive flow residual updates, and selectively smooth predictions according to confidence estimates. The architecture guarantees monotonic error reduction and robust sub-pixel convergence, even in scenarios with significant nonlinear radiation and geometric deformations.

Figure 1: The iterative discrepancy-guided flow refinement module leverages feature discrepancies and SGT to iteratively correct local misalignments across modalities.

The result is a geometrically consistent, dense registration field that is robust to both radiometric discrepancy and large-scale geometric transformation.

Experimental Results and Empirical Findings

Extensive experiments are conducted on representative cross-modal datasets: OSdataset (optical-SAR image pairs) and RoadScene (optical-infrared urban driving). CRFT is benchmarked against ten leading baselines, including both classical keypoint/handcrafted methods (e.g., HOWP, LNIFT, MSG, RIFT2), semi-dense transformers (e.g., XoFTR), and dense flow-based architectures (e.g., RAFT, GMFlow, GDROS, ADRNet).

Quantitative Outcomes:

On OSdataset, CRFT achieves an AEPE of 0.65 (subpixel accuracy, outperforming XoFTR+Flow by 42.5%) and CMR of 89.9% at a strict $0.7$ px threshold—significantly higher than all competitors (e.g., 41.7% for XoFTR+Flow, 35.5% for GDROS).

On RoadScene, CRFT yields the lowest AEPE (2.37) and highest CMR at 3 px (68.2%) and 1 px (18.2%), again surpassing the second-best deep learning (RAFT, ADRNet, GDROS) and classical techniques, which largely fail under modality disparities and strict accuracy criteria.

Figure 2: Visual comparisons demonstrate that CRFT produces highly consistent and geometrically robust flow fields aligning with annotated ground truth, superior to competing methods in multimodal scenarios.

Figure 3: The predicted geometric correspondences by CRFT result in the densest and most spatially coherent inliers across challenging cross-modal and affine-deformed cases.

Figure 4: Ablation analysis shows the critical impact of each architectural component, with the discrepancy-guided optimizer and iterative loss yielding substantial sub-pixel accuracy improvements.

Notably, CRFT is the only evaluated technique attaining both high coarse-to-fine stability and strong robustness at strict thresholds, confirming the necessity of the combined modality-independent, discrepancy-guided geometrical reasoning.

Comparative Analysis, Ablation, and Efficiency

Ablation studies further demonstrate the necessity of each component. Adding dense L1 supervision stabilizes learning, coarse flow estimation improves relaxed-threshold accuracy, whereas the DGFO and SGT modules produce order-of-magnitude gains in sub-pixel precision (increasing CMR at 0.7 px from 73.3% to 93.1%).

Model efficiency is also retained: CRFT contains 11.96M parameters, requires just 4.0 training hours on OSdataset, and each recurrent optimizer iteration introduces minimal additional cost (<1M params, ~0.10ms).

Qualitative Analysis and Robustness

Qualitative results demonstrate that CRFT delivers smooth, geometrically coherent flow fields, consistent checkerboard registration, and visually precise alignment even in regions with severe contrast, blur, or non-rigid distortion.

Figure 5: CRFT's predicted flow fields closely align with ground-truth displacements and present minimal error artifacts, evidencing robust sub-pixel registration under cross-modal gaps.

Figure 6: Stable and accurate behavior persists across the visually and radiometrically diverse RoadScene benchmark.

Figure 7: Overlayed visualizations of predicted and ground-truth registration quadrilaterals showcase CRFT's superior geometric accuracy, particularly under large deformations and modality shifts.

Figure 8: Checkerboard fusions validate the model's capacity to maintain fine-scale alignment of boundaries and textures across modalities.

Implications and Future Directions

CRFT establishes new state-of-the-art baselines for multimodal and cross-affine image registration, with theoretical and practical relevance for remote sensing, autonomous navigation, and medical imaging. The unified handling of global and local matching, the explicit decoupling of modality-dependent and geometric factors, and the recurrent, confidence-calibrated refinement offer a template for future work—especially for large-scale multimodal foundation model development and the design of invariant representations under extreme affine/radiometric perturbation.

While CRFT exhibits high robustness, failure is noted under combined extreme rotation and scale changes, suggesting fertile ground for further inclusion of invariant feature learning and generative modeling techniques.

Conclusion

CRFT delivers a significant advancement in cross-modal image registration by integrating a coarse-to-fine, transformer-based feature flow paradigm with discrepancy-driven geometric reasoning. The framework realizes both stable global alignment and precise local correspondence in heterogeneous domains, outperforming all major baselines in both empirical metric and visual quality. The model's architecture generalizes beyond registration, providing a modular backbone for next-generation multimodal correspondence and spatial reasoning tasks (2604.05689).