Multi-Scale Progressive Fusion Network for Single Image Deraining (2003.10985v2)

Abstract: Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image rain streak removal. For similar rain streaks at different positions, we employ recurrent calculation to capture the global texture, thus allowing to explore the complementary and redundant information at the spatial dimension to characterize target rain streaks. Besides, we construct multi-scale pyramid structure, and further introduce the attention mechanism to guide the fine fusion of this correlated information from different scales. This multi-scale progressive fusion strategy not only promotes the cooperative representation, but also boosts the end-to-end training. Our proposed method is extensively evaluated on several benchmark datasets and achieves state-of-the-art results. Moreover, we conduct experiments on joint deraining, detection, and segmentation tasks, and inspire a new research direction of vision task-driven image deraining. The source code is available at \url{https://github.com/kuihua/MSPFN}.

• The paper introduces MSPFN, a novel network that fuses coarse and fine features via multi-scale representations for enhanced rain removal.
• The approach incorporates convolutional LSTM and attention modules to capture global textures and refine channel-specific details effectively.
• Empirical evaluations demonstrate state-of-the-art PSNR and SSIM improvements, benefiting downstream tasks like object detection and segmentation.

Multi-Scale Progressive Fusion Network for Single Image Deraining

In the domain of computer vision, addressing the challenge of rain streak removal from images has significant implications for various applications, including object detection and semantic segmentation. The presented paper introduces the Multi-Scale Progressive Fusion Network (MSPFN), a novel framework designed to enhance single image deraining by leveraging multi-scale representation and fusion techniques.

Framework and Methodology

The MSPFN framework utilizes a multi-scale approach to capitalize on the inherent correlations of rain streaks at varying scales. This method is based on a pyramid structure, which allows for the integration of features from different resolutions. The process begins by generating Gaussian pyramid images of the input rain image, enabling the network to exploit complementary information across scales.

Core Modules:

1. Coarse Fusion Module (CFM): This module employs residual recurrent units with Conv-LSTM to capture global texture information at each scale, facilitating the representation of rain streaks through recurrent computation.
2. Fine Fusion Module (FFM): Incorporating channel attention units, the FFM refines the integration of multi-scale features by focusing on the most informative channels, thereby enhancing the discriminative power of the network.
3. Reconstruction Module (RM): This final module aggregates the processed multi-scale features to reconstruct the rain-free image, utilizing information from both the CFM and FFM.

The integration of these modules within the MSPFN facilitates progressive fusion, enabling the network to learn both coarse and fine details effectively, thereby improving the accuracy of rain removal.

Empirical Evaluation

The authors evaluate MSPFN's performance across several benchmark datasets, evidencing state-of-the-art results. Notably, the method demonstrates superior quantitative metrics such as PSNR and SSIM. Additionally, the practical utility of MSPFN is showcased in downstream tasks like object detection and segmentation, using datasets such as BDD and COCO. The MSPFN consistently outperforms comparable methods in these tasks, indicating its robustness and adaptability.

Contributions and Implications

This work advances the field of image deraining by:

• Introducing a multi-scale fusion strategy that leverages input image scales and deep hierarchical features in a coherent manner.
• Proposing an attention-based mechanism to fine-tune the integration of multi-scale information, enhancing the network's discriminative capacity.
• Demonstrating the framework's applicability to joint deraining and other vision tasks, highlighting the potential for further research into vision task-driven deraining approaches.

Future Prospects

The proposed MSPFN serves as a foundational model for ongoing research in adaptive image enhancement techniques. Further exploration into optimizing computational efficiency and extending the model's applicability to other adverse weather conditions can pave the way for more robust and efficient implementations in real-world scenarios, such as autonomous vehicles and surveillance systems.

In summary, this paper presents a well-structured and effective approach to tackle the complex issue of single image deraining, providing a significant contribution to both theoretical understanding and practical deployment within the field of computer vision.