FMRFusion: Frequency-Aware Multi-View Representation Learning for Heterogeneous Image Fusion

Abstract: Infrared and visible image fusion aims to generate a composite image that retains significant target information and preserves detailed textures, integrating two heterogeneous modalities. Previous image fusion methods typically adopt a single-module stacking approach to extract features from the two modalities. However, these approaches may result in incomplete learning of their distinct characteristics, thereby limiting the fusion effectiveness and constrain ing robustness in real-world heterogeneous data scenarios. To address these challenges, we propose FMRFusion, a frequency-aware multi-view representation learning network for Heterogeneous Image Fusion. A Multi-Scale Struc tural Perception Module is introduced to effectively capture discriminative structures, extracting fine-grained local structures and essential contextual information. A bilinear frequency decomposition mechanism is employed to sepa rate features into high-frequency and low-frequency components, enabling joint modeling of local details and global representations across different frequency domains. Moreover, a Cross-View Complementary Interaction is incorpo rated to explicitly model and fuse the complementary characteristics between reflected light information and radiative intensity responses, facilitating effective cross-view interaction. We further improve the Performance of the fused results by flow matching, which progressively refines the fused features by learning the transformation from coarse data to high-quality representations. Extensive experiments conducted on multiple benchmark datasets demonstrate that FMRFusion achieves superior and consistent performance across a range of fusion tasks, especially in nighttime scenarios

• The paper introduces a dual-branch CNN-Transformer architecture that fuses infrared and visible images using frequency-aware decomposition and cross-view interaction.
• It employs multi-scale structural perception and flow-matching refinement to preserve both modality-specific details and shared context.
• Experimental results on diverse datasets show notable improvements in entropy, contrast, texture, and edge sharpness compared to state-of-the-art methods.

Frequency-Aware Multi-View Representation Learning for Heterogeneous Image Fusion: FMRFusion

Introduction and Motivation

FMRFusion addresses key challenges in heterogeneous image fusion, specifically the integration of infrared and visible modalities. Traditional approaches often leverage single-path architectures and uniform feature extraction strategies, resulting in compromised modality-specific representation and suboptimal fusion quality. This work introduces a frequency-aware multi-view representation learning pipeline, combining dual-branch hybrid architecture (CNN + Transformer), explicit frequency decomposition, cross-view complementary interaction, and flow-matching refinement. These synergistic mechanisms jointly tackle incomplete feature decoupling, local-global tradeoff, and fusion mechanism limitations observed in prior CNN, AE, GAN, and Transformer-based models.

Methodology

The FMRFusion architecture comprises several specialized modules:

• Dual-Branch Hybrid Encoder: Independent branches for infrared and visible modalities employ SDE, TRB, and MS-SPM for shallow feature extraction, context modeling, and gradient-aware detail capture. The DRSformer block adaptively preserves salient self-attention features via learned sparsity.
• Multi-Scale Structural Perception Module (MS-SPM): Integrates Enhanced Attention Block (EAB) with large convolutional kernels and Gradient-Aware Block (GAB) employing Sobel filters for robust structural and detail feature extraction.

  Figure 1: Visual comparison of ``2'' from the TNO dataset.

• Bilinear Frequency Decomposition: Separates features into low-frequency (shared structural context) and high-frequency (modality-specific details) components, facilitating decoupled modeling of thermal vs. texture information.
• Cross-View Complementary Interaction (CVCI): Utilizes Lighten Cross-Attention (LCA), Intensity Enhancement Layer (IEL), and Color Denoise Layer (CDL) for explicit mutual refinement between infrared structural cues and visible chromatic/texture attributes.
• Flow-Matching Refinement: Employs semantic-guided interpolation from Gaussian priors to fusion representation, leveraging a conditional U-Net to iteratively optimize fusion quality under self-supervised constraints.

Loss Design & Training

A three-stage loss function architecture is adopted:

• Stage I: Feature decomposition constraints, gradient preservation, and reconstruction losses using SSIM and Euclidean norms.
• Stage II: Intensity preservation via element-wise maximum mapping from both sources, reinforcing saliency.
• Stage III: Flow-matching loss, semantic guidance, intensity, and gradient preservation. The system dynamically balances these via hyperparameters ($\alpha$, $\lambda$), ensuring progressive enhancement and task-adaptive optimization.

Experimental Results

Infrared-Visible Fusion Benchmarks

Extensive evaluations on MSRS, TNO, RoadScene, and M3FD datasets demonstrate the efficacy of FMRFusion:

• MSRS Dataset: FMRFusion yields highest EN, SD, SF, and AG, indicating superior information density, contrast, texture detail, and edge sharpness.
• TNO Dataset: Consistently best or second-best across EN, SD, SF, VIF, $Q^{AB/F}$, AG, outperforming strong baselines such as CDDFuse and SwinFusion.
• RoadScene and M3FD: Highest scores across SF, AG, EN, SD, VIF, confirming robust performance under complex and diverse scenarios.

Qualitative Analysis

Visual comparisons (Figures referenced) reveal that FMRFusion effectively integrates foreground thermal saliency and background texture, providing natural, artifact-free fusion with enhanced clarity and contrast.

Ablation Studies

Component ablations affirm the necessity of EAB, GAB, DRSformer, and CVCI; removing each results in measurable degradation across all quantitative metrics. Hyperparameter tuning of SDE sparsity parameter K shows optimal performance in [$\frac{1}{2}$, $\frac{4}{5}$]. Stage-wise training ablation confirms three-stage gradient flow as essential for peak performance.

Cross-Task Generalization

Medical Image Fusion

Direct transfer to MRI-CT and MRI-PET datasets produces best or second-best scores in EN, SD, SF, VIF, AG, validating domain-general dual-branch and three-stage learning design. FMRFusion preserves histological and functional features critical for diagnostic robustness.

Multifocus Image Fusion

On Lytro and RealMFF, FMRFusion surpasses DIFNet, CUNet, SDNet, FILM, etc., particularly in entropy, standard deviation, and average gradient. Both qualitative and quantitative benchmarks demonstrate strong clarity, contrast enhancement, and natural focus transitions, even in challenging scenes.

High-Level Vision Tasks: Object Detection

Fusion-driven detection with YOLOv5 on MSRS achieves highest [email protected] and [email protected]:0.95 for "Person" and "Car" classes. FMRFusion preserves structural infrared targets and visible spatial cues for enhanced confidence and localization, surpassing single-modality and multi-modal baselines.

Implications and Future Directions

FMRFusion sets a new standard for frequency-aware multi-view fusion, offering robust architecture and loss design that enable cross-modal decoupling and task-adaptive feature learning. The dual-branch decomposition, cross-attention, and flow-matching modules are compatible with vision-language and diffusion paradigms, suggesting applicability to autonomous systems, medical diagnostics, and industrial inspection. The architecture supports flexible extension to additional modalities (e.g., hyperspectral, depth).

Future developments may include unsupervised or self-supervised domain adaptation, transformer-based scalability optimizations, and real-time deployment strategies for video and temporal fusion. Further investigation is warranted on leveraging large-scale vision-LLMs for semantic-aware fusion, leveraging generative pipelines for data augmentation and zero-shot generalization.

Conclusion

FMRFusion introduces a comprehensive frequency-aware multi-view representation learning framework for heterogeneous image fusion, integrating CNN-Transformer dual-branch architecture, explicit frequency decomposition, cross-view complementary interaction, and flow-matching refinement. Extensive empirical validation across diverse fusion tasks, including infrared-visible, medical, and multi-focus datasets, verifies its strong quantitative and qualitative performance, robust task generalization, and superior high-level vision applicability. The system advances theoretical understanding and practical deployment prospects for cross-modal fusion in AI.