CrossFuse: Infrared and Visible Image Fusion

• CrossFuse is a framework that fuses infrared and visible images using specialized augmentation and frequency-aware networks.
• It employs dual approaches—multi-view augmentation with Top-K alignment and cross-attention with two-stage training—to optimize feature complementarity.
• Experimental evaluations show significant improvements in detail, contrast, and out-of-distribution robustness over previous methods.

CrossFuse refers to two distinct, state-of-the-art frameworks for infrared and visible image fusion, each addressing the multimodal complementarity and robustness challenges that arise in fusing IR/VI imagery for applications such as autonomous driving and surveillance. Both approaches are characterized by advanced feature extraction and fusion networks, optimization strategies tailored for cross-modal alignment or complementarity, and robust experimental performance. The term "CrossFuse" therefore denotes a family of leading techniques for infrared-visible fusion (Shi et al., 20 Feb 2025, Li et al., 2024).

1. Architectural Overview

Both CrossFuse frameworks share the goal of generating fused images that leverage complementary information from IR and visible inputs, while minimizing redundancy and addressing modality-specific gaps.

• CrossFuse (Multi-View Augmentation, Top-K Vision Alignment) (Shi et al., 20 Feb 2025): Employs a multi-view augmentation module comprising:
  • External-view augmentation: Cross-sensor Top-K selective vision alignment using an auxiliary visible dataset (M3FD) harmonized to the target visible distribution (RoadScene) via channel-wise gamma correction.
  • Internal-view augmentation: Weak-aggressive pipeline (random crop followed by Gaussian blur) for self-supervised contrastive consistency.
  • Frequency-Aware Fusion Network: Features are extracted using Restormer blocks (shallow), separated into low-frequency (Transformer) and high-frequency (CNN) branches, then fused with dedicated modules for each spectral band before being decoded for the final output.
• CrossFuse (Cross-Attention Mechanism, Two-Stage Training) (Li et al., 2024): Adopts a two-stage pipeline:
  • Stage 1: Two independent auto-encoders (IR/VI) trained on respective modalities to ensure strong within-modality feature extraction.
  • Stage 2: Features from the frozen encoders are fused using a novel cross-attention mechanism (CAM) involving self-attention (SA), a spatial "shift"/"unshift" operation, and cross-attention (CA) blocks with a reversed softmax (“re-softmax”) to promote complementary (uncorrelated) feature integration. The fused representation is decoded via a network with multi-level skip connections.

2. Cross-Sensor Alignment and Complementarity

• Top-K Selective Vision Alignment (Shi et al., 20 Feb 2025): Handles the negative impact of distribution mismatch when leveraging auxiliary visible datasets in IR/VI fusion. The algorithm selects the Top-K external patches most similar in RGB-channel statistics to each target patch, applies channel-wise gamma correction to match target histogram distributions, and thus augments training while minimizing domain gaps. Histograms of aligned patches overlay closely with the target after transformation, as demonstrated empirically.
• Cross-Attention for Complementarity (Li et al., 2024): Proposes a cross-attention mechanism that emphasizes uncorrelated, hence complementary, feature content. The re-softmax operation in CA promotes attention toward features with low inter-modality similarity, making the fused output rich in complementary, non-redundant structures. Self-attention and "shift"/"unshift" operations enrich context and preserve spatial coherence, leading to robust detail and contrast in fusion outputs.

3. Training Methodologies and Losses

• Multi-Stage Objectives:
  • (Shi et al., 20 Feb 2025) utilizes a two-stage training schedule:
  • Stage I: Feature reconstruction (MSE + SSIM) per modality and decomposition loss for frequency-band decorrelation.
  • Stage II: Fusion objective comprising decomposition, pixel- and gradient-level similarity, and a self-supervised consistency loss:

  $$\mathcal{L}_{ssl} = \theta(t)\,\|F_w - F_a\|_2^2, \quad \theta(t) = \theta_{init} \frac{1}{2}(\cos(\pi m / M)+1)$$

  where $m$ is the iteration, $M$ total iterations, and $\theta_{init}$ is the initial weight. - (Li et al., 2024) also adopts a two-stage approach: - Stage 1: Modality-specific auto-encoder reconstruction optimized as

  $$L_{auto}^c = \|I_c - I_c^r\|_F^2 + w_s\,\mathrm{SSIM}(I_c, I_c^r), \quad w_s=10^4$$ - Stage 2: CAM+decoder trained with intensity and gradient losses using adaptive modality masks to emphasize dominant local structures.

• Ablative Findings (Li et al., 2024):

  • The inclusion of SA/CA blocks, re-softmax, and shift operations each significantly boosts fusion metrics (Entropy, Standard Deviation, Mutual Information).

4. Experimental Evaluation and Performance

Experimental performance is assessed using comprehensive benchmarks and metrics:

<table> <thead> <tr><th>Dataset</th><th>Best Metrics for CrossFuse</th><th>Reference</th></tr> </thead> <tbody> <tr> <td>MSRS</td> <td>EN=6.63, MI=2.29, SD=44.14, SF=12.51, AG=3.96, VIF=0.90, Qabf=0.63, SSIM=0.95</td> <td>(Shi et al., 20 Feb 2025)</td> </tr> <tr> <td>RoadScene</td> <td>EN=7.44, MI=2.41, VIF=0.69, SCD=1.61, Qabf=0.59, SSIM=0.96</td> <td>(Shi et al., 20 Feb 2025)</td> </tr> <tr> <td>TNO (OOD)</td> <td>EN=7.21, SD=49.11, AG=5.49, VIF=0.77, Qabf=0.54, SSIM=0.99</td> <td>(Shi et al., 20 Feb 2025)</td> </tr> <tr> <td>TNO (CAM)</td> <td>EN=6.84, SD=73.47, MI=13.68, FMI_dct=0.39, SCD=1.77, SSIM=0.90</td> <td>(Li et al., 2024)</td> </tr> <tr> <td>VOT-RGBT (CAM)</td> <td>EN=6.89, SD=77.18, MI=13.78, FMI_dct=0.38, SCD=1.66, SSIM=0.91</td> <td>(Li et al., 2024)</td> </tr> </tbody> </table>

On the TNO OOD dataset, (Shi et al., 20 Feb 2025) demonstrates minor performance drop (<5% in spatial frequency) relative to large degradations among prior approaches. (Li et al., 2024) matches or outperforms peers in Entropy (EN), Standard Deviation (SD), and Mutual Information (MI), confirming state-of-the-art efficacy.

5. Robustness and Out-of-Distribution Generalization

CrossFuse (Shi et al., 20 Feb 2025) is explicitly designed for robust generalization to out-of-distribution data. Top-K aligned augmentation with cross-sensor adaptation and self-supervised internal consistency yield models that retain high performance across unseen domains. For example, in distribution shift experiments, CrossFuse maintains spatial frequency while DenseFuse drops by up to 21%. On TNO, which is never seen during training, CrossFuse surpasses all baselines in complementary-information metrics such as MI, VIF, and SSIM.

The CrossFuse approach based on cross-attention (Li et al., 2024) ensures robust detail and complementarity, with ablations showing that its re-softmax-based CAM and intra-modality preservation guard against redundant fusion and facilitate information transfer even when IR/VI content contrast is high.

6. Algorithmic Summary and Implementation Details

(Shi et al., 20 Feb 2025): Training Pseudocode

Initialize network parameters Θ, set hyper-params α1…α4, λ, μ, θ_init for epoch in 1…E: for each mini-batch: # External-view augmentation Sample target visible patches T Sample external visible patches E, compute diff(E,T) Keep Top-K E*, apply channel-wise gamma transform G(·) # Stage I: Feature reconstruction Encode I_ir, T ∪ G(E*) through shallow + L/H extractors Decode to reconstruct I'_ir, I'_vis Compute L_ir_rec, L_vis_rec, L_dec^{I} if epoch ≤ Stage I_epochs: Backpropagate L_total^{rec}, update Θ continue # Stage II: Fusion and internal augmentation Fuse X_ir^{L,H}, X_vis^{L,H} via F_L, F_H ⇒ I_fuse F_w = weak(I_fuse), F_a = aggr(F_w) Compute L_sim, L_ssl Backpropagate L_total^{fus}, update Θ

(Li et al., 2024): Stagewise Training

1. Train two auto-encoders (IR/VI) end-to-end per-modality until convergence.
2. Freeze encoders, pass IR/VI images → deep features.
3. Fuse features using SA/shift/SA/unshift/CA stack (CAM).
4. Decode to recover the fused image using skip connections.
5. Optimize intensity and gradient-based losses, using adaptive masks to separate IR/VI-dominant regions.

7. Contributions and Implications

• CrossFuse (Shi et al., 20 Feb 2025) advances data-centric IR/VI fusion for open-world system deployment by addressing both cross-sensor and cross-distribution shifts. The explicit OOD focus, external and internal augmentation strategies, and frequency-aware fusion network collectively yield improved image quality, detail preservation, and robustness across challenging conditions.
• CrossFuse (Li et al., 2024) establishes a new direction in feature fusion by formalizing complementarity as an optimization target via a re-softmax cross-attention mechanism and demonstrates that shift-based intra-modality self-attention exploits spatial structure for effective fusion.

These frameworks significantly expand the methodological toolkit for robust, high-fidelity multimodal fusion in real-world scenarios and set new state-of-the-art baselines for challenging datasets (Shi et al., 20 Feb 2025, Li et al., 2024).