DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion (2003.09210v3)

Abstract: Infrared and visible image fusion, a hot topic in the field of image processing, aims at obtaining fused images keeping the advantages of source images. This paper proposes a novel auto-encoder (AE) based fusion network. The core idea is that the encoder decomposes an image into background and detail feature maps with low- and high-frequency information, respectively, and that the decoder recovers the original image. To this end, the loss function makes the background/detail feature maps of source images similar/dissimilar. In the test phase, background and detail feature maps are respectively merged via a fusion module, and the fused image is recovered by the decoder. Qualitative and quantitative results illustrate that our method can generate fusion images containing highlighted targets and abundant detail texture information with strong robustness and meanwhile surpass state-of-the-art (SOTA) approaches.

• The paper introduces a novel deep image decomposition model within an auto-encoder framework to integrate background and detail features.
• It employs a bespoke loss function balancing similarity in background features with dissimilarity in details to optimize fusion performance.
• Evaluations on TNO, FLIR, and NIR datasets demonstrate enhanced target visibility, texture richness, and improved metrics over existing methods.

Overview of DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion

The paper "DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion" presents a novel approach for the fusion of infrared and visible images, leveraging advancements in deep learning methodologies, specifically auto-encoder (AE) networks. The primary objective of the proposed DIDFuse method is to produce an integrated output image that preserves the advantageous features of the source images while ensuring enhanced target recognition and detail precision.

Key Contribution

This research introduces a deep image decomposition model that operates within an AE framework, wherein both fusion and decomposition are realized entirely through neural network operations, specifically the encoder and decoder components of the AE. Compared to traditional methods that rely on manual filters and optimization techniques for decomposition, DIDFuse employs a purely data-driven strategy integrated into the deep learning paradigm.

1. Image Decomposition and Fusion: The encoder in DIDFuse is tasked with decomposing the input images into background and detail feature maps carrying low- and high-frequency information respectively. The subsequent fusion process necessitates manipulation within these feature maps to ensure feature amalgamation rather than simple pixel-wise addition.
2. Loss Function Design: The loss function is configured to balance the similarity in background feature maps and dissimilarity in detail feature maps. This strategic configuration aims to extract distinct thermal radiation and gradient details from the infrared and visible images.
3. Evaluation Methodology: Evaluations are performed across three datasets (TNO, FLIR, and NIR), consisting of varied environments and illumination conditions. Results indicate superior performance relative to state-of-the-art image fusion models, demonstrated by metrics such as entropy (EN), mutual information (MI), spatial frequency (SF), and visual information fidelity (VIF).

Empirical Results

Qualitative analysis of the fused images shows prominently highlighted targets and textured richness that consistently surpass existing fusion methods like FusionGAN, Densefuse, ImageFuse, among others. Quantitative performance further attests to its robustness across multiple datasets, showcasing strong reproducibility of results upon repetitive model training and testing.

Implications and Future Directions

The DIDFuse framework delivers significant implications for practical applications such as surveillance, military operations, and search-and-rescue operations where enhanced image clarity and target visibility are crucial. Theoretically, this work introduces promising directions for the integration of image decomposition and fusion within a unified deep learning model thus showing potential for further research in cognitive image processing.

Future research could extend the scope of DIDFuse by exploring alternative neural architectures and fusion strategies, as well as adapting this methodology to multi-modal and hyperspectral image data for enriched scene understanding. Additionally, optimizing the computational efficiency of the network could enable its deployment for real-time applications in constrained environments.

The paper provides a comprehensive exploration of DIDFuse as a pioneering approach in the fusion of infrared and visible images using deep neural networks. It sets a precedent for approaching image fusion tasks through holistic neural solutions, marking a step forward in the evolution of deep learning applications in graphical data processing.