EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment

Abstract: Evaluation is essential in image fusion research, yet most existing metrics are directly borrowed from other vision tasks without proper adaptation. These traditional metrics, often based on complex image transformations, not only fail to capture the true quality of the fusion results but also are computationally demanding. To address these issues, we propose a unified evaluation framework specifically tailored for image fusion. At its core is a lightweight network designed efficiently to approximate widely used metrics, following a divide-and-conquer strategy. Unlike conventional approaches that directly assess similarity between fused and source images, we first decompose the fusion result into infrared and visible components. The evaluation model is then used to measure the degree of information preservation in these separated components, effectively disentangling the fusion evaluation process. During training, we incorporate a contrastive learning strategy and inform our evaluation model by perceptual scene assessment provided by a LLM. Last, we propose the first consistency evaluation framework, which measures the alignment between image fusion metrics and human visual perception, using both independent no-reference scores and downstream tasks performance as objective references. Extensive experiments show that our learning-based evaluation paradigm delivers both superior efficiency (up to 1,000 times faster) and greater consistency across a range of standard image fusion benchmarks. Our code will be publicly available at https://github.com/AWCXV/EvaNet.

• The paper presents EvaNet, a multi-branch framework that efficiently approximates multiple IVIF metrics, achieving up to 1000× faster evaluations than traditional methods.
• It leverages modality decomposition, contrastive learning, and environment-aware adaptation to tailor metric predictions to scene-specific conditions and modality imbalances.
• Empirical results show significant improvements in metric consistency with both perceptual and downstream task rankings across several standard IVIF benchmarks.

Efficient and Consistent Evaluation for Infrared and Visible Image Fusion: An Analysis of EvaNet

Introduction

The evaluation of infrared and visible image fusion (IVIF) remains problematic due to the reliance on metrics originally designed for disparate computer vision tasks, resulting in issues of consistency and computational inefficiency. "EvaNet: Towards More Efficient and Consistent Infrared and Visible Image Fusion Assessment" (2604.02896) introduces a principled framework that addresses these limitations by leveraging a learning-based multi-branch architecture to approximate and adapt traditional metrics considering modality decomposition and environmental context. This essay offers a comprehensive analysis of the methodology, empirical findings, and implications of EvaNet, focusing on its approach to metric prediction, efficiency, and consistency across both reference-based and reference-free evaluations.

Motivation and Background

IVIF algorithms combine the complementary characteristics of visible and infrared imaging, benefiting downstream tasks such as detection and segmentation by enhancing perceptual scene understanding. However, established evaluation protocols typically adapt metrics (e.g., SSIM, FMI, VIF, PSNR) from unrelated domains, assuming equal contribution from each modality and neglecting scene-specific relevance (Figure 1).

Figure 1: Traditional image fusion metrics suffer from visual inconsistency and inefficiency; the evaluation may not correlate with perceived quality and can require significant computational time compared to the rapid inference of fusion models.

Further, the computational burden of evaluating multiple metrics sequentially restricts comprehensive benchmarking, often resulting in evaluation on only a subset of samples. The field lacks both metrics tailored for the IVIF context and systematic consistency evaluation referencing perceptual or downstream-task relevancy.

Figure 2: Evaluation pipelines show severe speed imbalance, with traditional metrics incurring up to $10^3 \times$ the evaluation time compared to model inference; EvaNet compresses all metric predictions into a single efficient forward pass.

EvaNet Architecture and Methodology

EvaNet pioneers a lightweight, multi-head evaluation network capable of synchronously predicting multiple fusion metrics by adopting a divide-and-conquer paradigm:

• Decomposition: The fused image is decomposed into infrared and visible components via pre-trained information probes, allowing independent assessment of each modality's information preservation.
• Three-Branch Structure: The architecture comprises infrared, visible, and environment branches. The environment branch incorporates LLM-derived cues (e.g., ChatGPT-4o) to adaptively penalize modality imbalance in challenging scenarios.
• Contrastive Learning: Positive pairs (original and recomposed images) and negative pairs (unrelated images) train the network to map perceptually and semantically consistent features to high metric scores while suppressing irrelevant associations (Figure 3).

  Figure 4: EvaNet architecture features modality-specific decomposition, dual similarity regressors, and an environment-aware regressor, all yielding metric predictions from a unified feature extraction backbone.

  

  Figure 3: During training, contrastive learning is employed—positive pairs of inputs are pushed close in the metric space, while negative (non-matching) pairs are pushed apart for enhanced discriminability.

• Environment-Aware Adaptation: LLMs annotate each scene's illumination and obscuration, generating an ENV penalty factor during training to dynamically weight or penalize the visible modality in adverse environmental conditions (Figure 5, Figure 6).

  Figure 5: Scene environment is assessed using LLMs, which provide normalized illumination and occlusion scores for each image pair, guiding the environment branch during training.

Notably, the architecture efficiently predicts $N$ metrics in a single GPU-accelerated forward pass, achieving up to $1000\times$ speed improvements over sequential CPU-based implementations.

Metric Consistency Protocol and Evaluation

EvaNet introduces a novel metric consistency (MC) criterion that calibrates the correlation between metric-based ranking and reference rankings derived from third-party models. Two protocols are instantiated:

• $\textrm{MC}_\textrm{deep}$: Uses deep no-reference image quality models (e.g., DeepIQA) as the perceptual reference.
• $\textrm{MC}_\textrm{ds}$: Uses downstream task models (object detectors, segmenters) as task-driven references.

Rank discrepancies are weighted by decaying penalties with higher importance assigned to top-ranking candidates, providing a more discriminative assessment of metric reliability relative to both human and task-centric criteria.

Empirical Results

Efficiency

Empirical benchmarks demonstrate that EvaNet achieves an order-of-magnitude speed-up: for 3463 fused images, EvaNet computes 8 metrics in under 11 seconds, compared to over 24 hours required using classic methods, independent of GPU/CPU speedup effects due to the change in evaluation paradigm.

Consistency

Across three standard IVIF benchmarks (LLVIP, RoadScene, TNO) and the MSRS generalization set, EvaNet consistently improves the MC scores for a spectrum of traditional metrics—including VIF, Qabf, FMI, and PSNR—against both original and TextAttn variants (Table 1). Gains are particularly marked for VIF and Qabf, which reflect information fidelity and salient feature preservation, and for reference-free metrics when retrained in the EvaNet framework.

EvaNet's superior consistency extends to subjective visual references (CLIP-IQA, Figure 7) and downstream task performance, aligning metric rankings closer to semantic segmentation and detection outputs (Figure 8, Figure 9). Ablation studies further establish the indispensability of each architectural component: environment awareness, contrastive learning, and decomposition all contribute positive, additive improvements (Table 2).

Interpretability and Generalization

EvaNet's metric-wise attribution maps demonstrate correspondence between learned attention and underlying metric semantics (Figure 10). Its environment-aware adaptation is robust across varying LLM supervisors (Figure 6), with advanced LLMs further improving consistency. Generalization experiments on unseen datasets (MSRS) and to reference-free metrics underscore its transferability.

Qualitative Analysis

Qualitative results verify that for cases where vanilla metrics fail to penalize artefacts or loss of modality-specific information, EvaNet's adapted metrics realign the scores, reflecting perceptual or functional superiority more reliably (Figure 11, Figure 12, Figure 13, Figure 14).

Limitations

EvaNet's consistency protocol relies on third-party models (for perceptual or task alignment). While objective and reproducible, this may not capture subjective or application-specific quality preferences comprehensively. Although designed for IVIF, the methodology generalizes conceptually to other fusion scenarios (e.g., multi-exposure), prompting future investigation into dynamic environment-aware weighting in broader contexts.

Implications and Future Directions

EvaNet establishes a scalable, efficient, and consistent evaluation paradigm for IVIF, opening multiple lines of future inquiry:

• Integration into Training Loops: By providing fast and reliable metric gradients, EvaNet could enable dynamic loss adaptation for fusion networks, shaping optimization towards perceptually coherent and task-relevant outputs.
• Context-Aware Metric Learning: The architecture and environment-branch affordances invite exploration into adaptive metric learning for other fusion types, such as multi-focus or multispectral fusion.
• Reference-Free and Unified Assessment: The approach demonstrates extensibility to reference-free quality assessment, potentially enabling unified evaluation pipelines for arbitrary fusion tasks.

Conclusion

EvaNet marks a significant advance in the assessment of IVIF results by combining fast, parallelizable metric prediction with context- and modality-aware adaptations, validated by a principled consistency protocol referencing perceptual and functional ground truths. Its contributions position it as a reference framework for efficient, interpretable, and reliable metric evaluation in both academic and applied multimodal image fusion research.