Deep Stacked Hierarchical Multi-patch Network for Image Deblurring (1904.03468v1)

Abstract: Despite deep end-to-end learning methods have shown their superiority in removing non-uniform motion blur, there still exist major challenges with the current multi-scale and scale-recurrent models: 1) Deconvolution/upsampling operations in the coarse-to-fine scheme result in expensive runtime; 2) Simply increasing the model depth with finer-scale levels cannot improve the quality of deblurring. To tackle the above problems, we present a deep hierarchical multi-patch network inspired by Spatial Pyramid Matching to deal with blurry images via a fine-to-coarse hierarchical representation. To deal with the performance saturation w.r.t. depth, we propose a stacked version of our multi-patch model. Our proposed basic multi-patch model achieves the state-of-the-art performance on the GoPro dataset while enjoying a 40x faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280x720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For stacked networks, significant improvements (over 1.2dB) are achieved on the GoPro dataset by increasing the network depth. Moreover, by varying the depth of the stacked model, one can adapt the performance and runtime of the same network for different application scenarios.

Deep Stacked Hierarchical Multi-patch Network for Image Deblurring

Overview

The paper "Deep Stacked Hierarchical Multi-patch Network for Image Deblurring" by Zhang et al. introduces a novel approach to address the challenges of non-uniform blind image deblurring. The authors propose a deep hierarchical multi-patch network inspired by Spatial Pyramid Matching (SPM), aimed at overcoming the limitations inherent in existing multi-scale and scale-recurrent models. These traditional models are burdened by high computational costs and performance saturation as a result of increased network depth.

Methodology

The proposed network leverages a fine-to-coarse hierarchical representation to process blurry images efficiently. The Deep Multi-Patch Hierarchical Network (DMPHN) consists of a basic multi-patch model that operates in a residual-like manner. This enables each network layer to focus on different scales of blur, leading to improved deblurring performance. The strategy diverges from traditional coarse-to-fine methods by reducing the need for large filter sizes and decreasing runtime, which allows for real-time performance.

Results

The DMPHN exhibits state-of-the-art performance on the GoPro dataset with a significant reduction in runtime. The basic model processes 1280x720 image resolution at 30ms, achieving real-time processing for 720p images at 30fps. The stacked version of the model, which involves stacking multiple DMPHNs, achieves over a 1.2dB improvement on the GoPro dataset by increasing network depth. The model size is considerably more efficient, being 10 times smaller than some competing models, which positions it favorably in terms of computational efficiency.

Implications

This research has significant implications for applications requiring real-time image processing, such as video streaming and mobile photography. The adaptability of performance and runtime by varying the depth of the stacked model allows it to cater to different application scenarios effectively. The proposed stacked architecture also suggests a new direction for enhancing the depth of deblurring models without encountering performance saturation.

Future Developments

The introduction of a stacked architecture could lead to further exploration in AI model optimization, not limited to image deblurring. Future research can investigate the broader applicability of similar architectures in other domains, such as video enhancement or image super-resolution.

Moreover, the potential for extending the hierarchical multi-patch approach to integrate with advanced architectures or leverage pre-trained models could further enhance its capabilities and efficiency. The exploration of weight-sharing techniques also presents a promising area for reducing model size without substantial loss in performance.

Conclusion

The hierarchical multi-patch network presented in this work provides a substantial contribution to the field of image deblurring, offering a robust yet computationally efficient solution to common hindrances faced by existing methodologies. Its ability to achieve high-quality deblurring in real-time frames it as a versatile tool with wide-ranging applications, setting a foundation for future research into stacked architectures in AI.