Intriguing Findings of Frequency Selection for Image Deblurring (2111.11745v2)

Abstract: Blur was naturally analyzed in the frequency domain, by estimating the latent sharp image and the blur kernel given a blurry image. Recent progress on image deblurring always designs end-to-end architectures and aims at learning the difference between blurry and sharp image pairs from pixel-level, which inevitably overlooks the importance of blur kernels. This paper reveals an intriguing phenomenon that simply applying ReLU operation on the frequency domain of a blur image followed by inverse Fourier transform, i.e., frequency selection, provides faithful information about the blur pattern (e.g., the blur direction and blur level, implicitly shows the kernel pattern). Based on this observation, we attempt to leverage kernel-level information for image deblurring networks by inserting Fourier transform, ReLU operation, and inverse Fourier transform to the standard ResBlock. 1x1 convolution is further added to let the network modulate flexible thresholds for frequency selection. We term our newly built block as Res FFT-ReLU Block, which takes advantages of both kernel-level and pixel-level features via learning frequency-spatial dual-domain representations. Extensive experiments are conducted to acquire a thorough analysis on the insights of the method. Moreover, after plugging the proposed block into NAFNet, we can achieve 33.85 dB in PSNR on GoPro dataset. Our method noticeably improves backbone architectures without introducing many parameters, while maintaining low computational complexity. Code is available at https://github.com/DeepMed-Lab/DeepRFT-AAAI2023.

• The paper presents the Res FFT-ReLU Block that fuses frequency domain manipulation with spatial convolution to capture blur kernels and enhance deblurring performance.
• Experimental results demonstrate measurable improvements in PSNR and SSIM, notably with a PSNR gain from 33.69 dB to 33.85 dB on the GoPro dataset.
• The study underscores the value of combining frequency and spatial features, paving the way for more efficient and robust image restoration techniques.

An Analysis of Frequency Selection in Image Deblurring

The paper "Intriguing Findings of Frequency Selection for Image Deblurring" presents an innovative perspective on image deblurring through leveraging frequency domain operations. This research addresses the complexity of image blurring, traditionally solved through spatial domain methods using end-to-end architectures that learn differences between blurry and sharp image pairs. The authors propose an alternative approach by highlighting the relevance of blur kernels via frequency selection.

Core Contribution

The principal contribution of this work is the introduction of a novel Res FFT-ReLU Block. This block integrates frequency domain manipulations into a standard residual block architecture to enhance the deblurring process by combining pixel-level spatial features with kernel-level frequency domain features. This integration is achieved by inserting Fast Fourier Transform (FFT), ReLU, and inverse FFT operations within the residual block framework, alongside typical convolutions.

Key Findings and Methodology

The paper begins by exploring the effect of applying a ReLU operation on the frequency domain of a blurred image, followed by an inverse Fourier Transform. It reveals that such manipulation can extract inherent blur patterns, including direction and intensity, which are otherwise overlooked in pixel-level approaches. This phenomenon motivates the design of the Res FFT-ReLU Block, which fuses frequency and spatial domain features to improve deblurring quality.

The proposed block introduces a flexible convolution layer following the FFT to adaptively select frequency components, modulating thresholds through learned parameters, thus enhancing deblurring performance.

Experimental Validation

The paper reports thorough experimental results demonstrating the efficacy of the proposed method. Upon integrating the Res FFT-ReLU Block into various architectures, such as NAFNet and MIMO-UNet, the network achieved significant improvements in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) across standard benchmarks like the GoPro, HIDE, and RealBlur datasets.

Particularly notable is the performance on the GoPro dataset, where the integration of the Res FFT-ReLU Block into the NAFNet architecture leads to a PSNR gain from 33.69 dB to 33.85 dB, while maintaining computational efficiency with relatively low model complexity in terms of parameters and FLOPs.

Implications and Future Directions

This work underscores the value of frequency domain information in restoring blurred images. The integration of frequency components with spatial learning enhances model robustness against diverse types of blur patterns, suggesting a promising direction for improving image restoration tasks comprehensively.

Moreover, the successful application of FFT-ReLU operations in capturing blur kernels could inspire further investigation into other frequency-based transforms or their hybrid adaptations for varied image processing tasks. Exploring more efficient models that balance computational cost with deblurring effectiveness could also be a valuable future endeavor.

In summary, the paper contributes a significant methodological advancement in image deblurring by synergizing frequency domain insights with traditional spatial operations, opening avenues for enhanced image restoration techniques in both academic research and real-world applications.