LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images (2304.05172v2)

Abstract: Deep learning based fusion methods have been achieving promising performance in image fusion tasks. This is attributed to the network architecture that plays a very important role in the fusion process. However, in general, it is hard to specify a good fusion architecture, and consequently, the design of fusion networks is still a black art, rather than science. To address this problem, we formulate the fusion task mathematically, and establish a connection between its optimal solution and the network architecture that can implement it. This approach leads to a novel method proposed in the paper of constructing a lightweight fusion network. It avoids the time-consuming empirical network design by a trial-and-test strategy. In particular we adopt a learnable representation approach to the fusion task, in which the construction of the fusion network architecture is guided by the optimisation algorithm producing the learnable model. The low-rank representation (LRR) objective is the foundation of our learnable model. The matrix multiplications, which are at the heart of the solution are transformed into convolutional operations, and the iterative process of optimisation is replaced by a special feed-forward network. Based on this novel network architecture, an end-to-end lightweight fusion network is constructed to fuse infrared and visible light images. Its successful training is facilitated by a detail-to-semantic information loss function proposed to preserve the image details and to enhance the salient features of the source images. Our experiments show that the proposed fusion network exhibits better fusion performance than the state-of-the-art fusion methods on public datasets. Interestingly, our network requires a fewer training parameters than other existing methods. The codes are available at https://github.com/hli1221/imagefusion-LRRNet

• The paper introduces a novel low-rank guided fusion architecture that optimizes representation learning for merging infrared and visible images.
• The LLRR framework decomposes images into low-rank and salient features, achieving superior detail preservation and enhanced infrared features.
• The network reduces computational overhead with an efficient detail-to-semantic loss, demonstrating robust performance on TNO and VOT2020-RGBT datasets.

LRRNet: A Novel Representation-Learning Guided Fusion Network for Infrared and Visible Images

This paper presents a novel approach to the image fusion problem, specifically targeting the integration of infrared (IR) and visible light images, using an advanced neural network architecture named LRRNet. The fusion of infrared and visible images is an essential task in computer vision, with applications ranging from autonomous vehicles to surveillance systems. The unique contribution of this paper is the development of a representation-learning guided framework to construct a lightweight network for image fusion, addressing both efficiency and performance issues traditionally associated with this challenge.

Key Contributions

1. Design of a LRR-based Lightweight Fusion Architecture: The authors introduce a mathematically grounded approach to designing the fusion network architecture. This involves framing the fusion task as an optimisation problem, with the solution directly influencing the network design. The use of a low-rank representation (LRR) model to guide the design process distinguishes this work from conventionally empirical methods.
2. Novel Learnable Representation Model: The LRRNet employs a new learnable low-rank representation model, termed LLRR, to decompose source images into low-rank and salient features. This model is instrumental in preserving image details and enhancing infrared features in the final fused image.
3. Efficient Fusion Strategy: The fusion strategy utilizes a combination of convolutional layers within the uniquely designed LLRR blocks, integrating the source images' features, which are then recombined to produce the fused image. This method reduces computational overhead by replacing iterative optimisation procedures with a forward-pass network.
4. Detail-to-Semantic Loss Function: The loss function designed for the training of LRRNet is comprehensive. It combines pixel-level, shallow, middle, and deep feature-level losses, enabling the network to retain important details from visible images while accentuating infrared features.

Results and Implications

The proposed LRRNet has been rigorously evaluated using the TNO and VOT2020-RGBT datasets, demonstrating superior performance in several key metrics, such as Entropy (En), Mutual Information (MI), and Standard Deviation (SD), compared to state-of-the-art methods. Notably, LRRNet not only shows improved performance in preserving image details and infrared features but also achieves this with a significantly reduced number of training parameters, attesting to its efficiency and suitability for real-world applications where computational resources may be limited.

Furthermore, LRRNet's architecture modularity and its successful application in an RGBT object tracking task underscore its potential adaptability to other multi-modal fusion scenarios. This flexibility implies that the principles detailed in this paper can influence future developments in other areas of artificial intelligence and computer vision, especially those requiring effective integration of diverse modalities.

Future Directions

While LRRNet marks a significant step forward in developing efficient deep learning-based fusion networks, several avenues for future research exist. First, exploring automated or semi-automated techniques for adjusting the complex parameters of the model and loss functions could enhance model applicability and ease of use. Second, testing the adaptability of LRRNet across diverse multi-modal datasets beyond infrared and visible spectrum images could further extend its applicability. Lastly, investigating the model’s potential roles in other application domains, such as medical imaging or enhanced human-computer interaction interfaces, could open new frontiers for research and practical deployment.

In conclusion, LRRNet offers a promising new direction for image fusion tasks, leveraging representation learning to inform network design and achieve high-quality fusion results while maintaining computational efficiency.