Clearing the Skies: A deep network architecture for single-image rain removal (1609.02087v2)

Abstract: We introduce a deep network architecture called DerainNet for removing rain streaks from an image. Based on the deep convolutional neural network (CNN), we directly learn the mapping relationship between rainy and clean image detail layers from data. Because we do not possess the ground truth corresponding to real-world rainy images, we synthesize images with rain for training. In contrast to other common strategies that increase depth or breadth of the network, we use image processing domain knowledge to modify the objective function and improve deraining with a modestly-sized CNN. Specifically, we train our DerainNet on the detail (high-pass) layer rather than in the image domain. Though DerainNet is trained on synthetic data, we find that the learned network translates very effectively to real-world images for testing. Moreover, we augment the CNN framework with image enhancement to improve the visual results. Compared with state-of-the-art single image de-raining methods, our method has improved rain removal and much faster computation time after network training.

• The paper's main contribution is a novel detail layer training strategy that simplifies single-image rain removal using the DerainNet architecture.
• It utilizes a three-layer CNN to map rainy detail layers to clean ones, yielding higher SSIM and lower BIQI scores compared to state-of-the-art methods.
• The method leverages synthetic training data to effectively generalize to real-world images, enabling fast computation suitable for real-time applications.

Overview of "Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal"

The paper "Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal" by Xueyang Fu, Jiabin Huang, Xinghao Ding, Yinghao Liao, and John Paisley presents a novel approach for the removal of rain streaks from single images using a convolutional neural network (CNN) named "DerainNet". This work is conducted at Xiamen University and Columbia University and is supported by major Chinese science foundations.

Background and Related Work

Rain removal methods are divided into video-based and single-image-based approaches. While video-based methods leverage temporal information to effectively detect rain streaks, single-image de-raining is significantly more challenging due to the lack of temporal redundancy. Previous single-image rain removal methods primarily rely on low-level features and dictionary learning approaches to decompose images into base and detail layers. However, these methods often struggle with preserving fine details and handling images with complex structures similar to rain streaks.

Contributions

The main contributions of this paper are:

1. Detail Layer Training Strategy: Instead of increasing network depth or breadth, the authors apply domain knowledge from image processing. They decompose images into base and detail layers and train the CNN on the detail layer, thereby simplifying the learning task and yielding better de-raining results.
2. Efficient Network Design: The network learns the nonlinear mapping between rainy and clean detail layers with minimal architecture, leading to improved deraining and visual quality enhancement.
3. Synthetic Training Data: Due to the unavailability of ground truth for real-world rainy images, the authors synthesize a training dataset. Despite training on synthetic data, DerainNet generalizes effectively to real-world scenarios.

Methodology

Network Architecture

DerainNet consists of a three-layer CNN with two hidden layers. The architecture is described by the following stages:

• Feature Extraction: The first layer extracts features from the high-frequency detail content of the input image.
• Nonlinear Transformation: The second layer performs rain streak removal by applying a nonlinear transformation.
• Image Reconstruction: The final layer reconstructs the clean image detail layer.

Training Strategy

The network is trained on a dataset of synthesized rainy images. The image is first decomposed into base and detail layers using guided filtering. The network is trained to learn the mapping from the rainy image's detail layer to the clean image's detail layer. This approach is validated by demonstrating faster convergence and improved rain removal.

Experimental Results

Synthetic and Real-World Data

The authors validate DerainNet on both synthetic and real-world rainy images. Quantitative metrics like the Structure Similarity Index (SSIM) are used for synthetic data, while the Blind Image Quality Index (BIQI) is used for real-world images. DerainNet outperforms other state-of-the-art methods in both domains, as corroborated by higher SSIM and lower BIQI scores.

Efficiency

DerainNet exhibits significantly faster computation times compared to existing methods, making it suitable for real-time applications. The efficient design of the network, leveraging image processing knowledge, allows for effective rain streak removal without the need for deep and computationally intensive networks.

Implications and Future Directions

The approach presented in this paper demonstrates the potential of integrating domain-specific knowledge with deep learning architectures. The ability to effectively train on synthetic data and generalize to real-world scenarios is particularly notable. Practically, this method can be applied in various fields such as autonomous driving, outdoor surveillance, and any other image processing tasks requiring robust de-raining algorithms.

Future exploration could include enhancing the network architecture to further improve accuracy and efficiency. Additionally, expanding the training dataset with more varied scenes and rain patterns could potentially improve the generalization capability of DerainNet. Further research may also consider exploring advanced image enhancement techniques to complement the high-frequency detail layer strategy used in this work.

In conclusion, "Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal" is a significant contribution to the field of image processing, offering a robust and efficient solution to the challenging problem of single-image rain removal.