Infrared and Visible Image Fusion using a Deep Learning Framework (1804.06992v4)

Abstract: In recent years, deep learning has become a very active research tool which is used in many image processing fields. In this paper, we propose an effective image fusion method using a deep learning framework to generate a single image which contains all the features from infrared and visible images. First, the source images are decomposed into base parts and detail content. Then the base parts are fused by weighted-averaging. For the detail content, we use a deep learning network to extract multi-layer features. Using these features, we use l_1-norm and weighted-average strategy to generate several candidates of the fused detail content. Once we get these candidates, the max selection strategy is used to get final fused detail content. Finally, the fused image will be reconstructed by combining the fused base part and detail content. The experimental results demonstrate that our proposed method achieves state-of-the-art performance in both objective assessment and visual quality. The Code of our fusion method is available at https://github.com/hli1221/imagefusion_deeplearning

• The paper introduces a novel deep learning method that leverages VGG-19 for multi-layer feature extraction to fuse infrared and visible images.
• It decomposes images into base and detail components, using weighted averaging and soft-max guided selection to preserve salient features.
• Quantitative evaluations show improved FMI, SSIM, and reduced noise/artifacts compared to traditional fusion methods.

Infrared and Visible Image Fusion using a Deep Learning Framework

The paper "Infrared and Visible Image Fusion using a Deep Learning Framework" by Hui Li et al. addresses the challenging problem of integrating infrared and visible images into a cohesive, informative representation. It introduces a novel deep learning-based approach to improve the performance of image fusion techniques by leveraging deep feature extraction, addressing the limitations of traditional methods that rely heavily on discrete transforms and sparse representations.

Methodology

The authors propose a fusion method leveraging a pre-trained VGG-19 convolutional neural network (CNN) to extract multi-layer features from the source images. The process begins with decomposing the input images into base and detail components. The base parts are fused using a straightforward weighted-averaging strategy. In contrast, the detail components benefit from a more sophisticated approach. The VGG-19 model extracts hierarchical feature maps, which are then processed to generate weight maps through a soft-max operation. These weight maps guide the selection of detail features to form the final fused detail content.

To reconstruct the final composite image, the fused detail content and base images are combined. This method aims to preserve the salient features of both modalities while ensuring that the fused image maintains high visual quality and structural integrity.

Technical Evaluation

Experiments performed on 21 pairs of infrared and visible source images demonstrate the effectiveness of the proposed method. The paper provides quantitative comparisons using metrics such as FMI (Feature Mutual Information) for discrete cosine and wavelet features, modified structural similarity (SSIM), and noise/artifact ratio (N_abf). The proposed deep learning-based fusion method not only surpasses existing methods like joint sparse representation (JSR) and convolutional sparse representation (ConvSR) in preserving feature detail but also in minimizing artificial noise and artifacts as evidenced by superior performance metrics, notably achieving a significant reduction in the N_abf values compared to contemporary methods.

Implications and Future Work

This research presents a robust framework that exploits the power of deep learning to enhance image fusion processes. By introducing a multi-layer feature extraction approach, this paper significantly contributes to the field by advancing the state-of-the-art in both theoretical and practical dimensions. The results suggest that the fusion method can be extended beyond infrared-visible image fusion to other domains, such as medical imaging, multi-exposure, and multi-focus imaging, where different image modalities or perspectives need to be seamlessly integrated.

Future research might explore augmenting the current architecture with more advanced network designs or transfer learning techniques to further enhance feature extraction and integration capabilities. Additionally, investigating adaptive fusion strategies based on the content of the source images and extending the model to support real-time applications are promising avenues for further paper. By doing so, this line of work not only expands the applicability of deep learning in image processing but also meets the growing demand for efficient, high-quality image fusion systems in various technological contexts.