Papers
Topics
Authors
Recent
Search
2000 character limit reached

F2former: When Fractional Fourier Meets Deep Wiener Deconvolution and Selective Frequency Transformer for Image Deblurring

Published 3 Sep 2024 in cs.CV | (2409.02056v1)

Abstract: Recent progress in image deblurring techniques focuses mainly on operating in both frequency and spatial domains using the Fourier transform (FT) properties. However, their performance is limited due to the dependency of FT on stationary signals and its lack of capability to extract spatial-frequency properties. In this paper, we propose a novel approach based on the Fractional Fourier Transform (FRFT), a unified spatial-frequency representation leveraging both spatial and frequency components simultaneously, making it ideal for processing non-stationary signals like images. Specifically, we introduce a Fractional Fourier Transformer (F2former), where we combine the classical fractional Fourier based Wiener deconvolution (F2WD) as well as a multi-branch encoder-decoder transformer based on a new fractional frequency aware transformer block (F2TB). We design F2TB consisting of a fractional frequency aware self-attention (F2SA) to estimate element-wise product attention based on important frequency components and a novel feed-forward network based on frequency division multiplexing (FM-FFN) to refine high and low frequency features separately for efficient latent clear image restoration. Experimental results for the cases of both motion deblurring as well as defocus deblurring show that the performance of our proposed method is superior to other state-of-the-art (SOTA) approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Defocus deblurring using dual-pixel data. In European Conference on Computer Vision, pages 111–126. Springer, 2020.
  2. Learning to reduce defocus blur by realistically modeling dual-pixel data. In Proc. ICCV, pages 2269–2278, 10 2021.
  3. End-to-end object detection with transformer. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  4. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  5. Simple baselines for image restoration. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
  6. Simple baselines for image restoration. arXiv preprint arXiv:2204.04676, 2022.
  7. Rethinking coarse-to-fine approach in single image deblurring. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4621–4630, 2021.
  8. Rethinking coarse-to-fine approach in single image deblurring. In Proc. ICCV, pages 4621–4630, 10 2021.
  9. Selective frequency network for image restoration. In The Eleventh International Conference on Learning Representations, 2023.
  10. Multi-scale residual low-pass filter network for image deblurring. In Proc. ICCV, pages 12311–12320, 10 2023.
  11. Deep wiener deconvolution: Wiener meets deep learning for image deblurring. Advances in Neural Information Processing Systems, 33:1048–1059, 2020.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
  13. Short-time fourier transform: two fundamental properties and an optimal implementation. IEEE Transactions on Signal Processing, 51(5):1231–1242, 2003.
  14. Self-supervised non-uniform kernel estimation with flow-based motion prior for blind image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18105–18114, 2023.
  15. Marie Farge. Wavelet transforms and their applications to turbulence. Annual Review of Fluid Mechanics, 24(1):395–458, 1992.
  16. Loukas Grafakos. Classical fourier analysis. 2010.
  17. Efficient frequency domain-based transformers for high-quality image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  18. Trainable fractional fourier transform. IEEE Signal Processing Letters, 31:751–755, 2024.
  19. Fast image deconvolution using hyper-laplacian priors. In Advances in Neural Information Processing Systems (NIPS), pages 1033–1041, 2009.
  20. Blind image deconvolution. IEEE Signal Processing Magazine, 13(3):43–64, 1996.
  21. Optimal filtering in fractional fourier domains. In 1995 International Conference on Acoustics, Speech, and Signal Processing, volume 2, pages 937–940 vol.2, 1995.
  22. Towards robust, locally linear deep networks. In International Conference on Learning Representations (ICLR), 2019.
  23. Deep defocus map estimation using domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  24. Iterative filter adaptive network for single image defocus deblurring. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2034–2042, 2021.
  25. Selective kernel networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 510–519, 2019.
  26. Efficient and explicit modelling of image hierarchies for image restoration. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18278–18289, 2023.
  27. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021.
  28. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  29. Deep residual fourier transformation for single image deblurring. ArXiv, abs/2111.11745, 2021.
  30. Intriguing findings of frequency selection for image deblurring. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 2023.
  31. Deep multi-scale convolutional neural network for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  32. Digital computation of the fractional fourier transform. IEEE Transactions on Signal Processing, 44(9):2141–2150, 1996.
  33. The Fractional Fourier Transform: With Applications in Optics and Signal Processing. Wiley, New York, NY, USA, 2001.
  34. Robust kernel estimation with outliers handling for image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2800–2808, 2016.
  35. Phase-only image based kernel estimation for single image blind deblurring. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6027–6036, 2019.
  36. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training. In Proc. ECCV, pages 327–343, 11 2020.
  37. Microscopy image restoration with deep wiener-kolmogorov filters. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, page 185–201, Berlin, Heidelberg, 2020. Springer-Verlag.
  38. Fractional fourier transform phase for image matching. pages 688–697, 01 2022.
  39. Real-world blur dataset for learning and benchmarking deblurring algorithms. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  40. Revisiting image deblurring with an efficient convnet, 2023.
  41. Human-aware motion deblurring. In IEEE International Conference on Computer Vision, 2019.
  42. Single image defocus deblurring using kernel-sharing parallel atrous convolutions. In Proc. ICCV, 2021.
  43. Stripformer: Strip transformer for fast image deblurring. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
  44. Uformer: A general u-shaped transformer for image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  45. N. Wiener. Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications, volume 113. MIT Press, 1949.
  46. Adarevd: Adaptive patch exiting reversible decoder pushes the limit of image deblurring. In Proc. CVPR, 2024.
  47. Two-phase kernel estimation for robust motion deblurring. In Proceedings of the European Conference on Computer Vision (ECCV), volume 1, pages 157–170, 2010.
  48. Multi-class token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  49. Motion blur kernel estimation via deep learning. IEEE Transactions on Image Processing, 27(1):194–205, 2018.
  50. Image restoration based on the discrete fraction fourier transform. Proceedings of SPIE - The International Society for Optical Engineering, 09 2001.
  51. Y-net: Multi-scale feature aggregation network with wavelet structure similarity loss function for single image dehazing. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2628–2632. IEEE, 2020.
  52. Deep fractional fourier transform. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.
  53. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, 2022.
  54. Multi-stage progressive image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  55. A.I. Zayed. A convolution and product theorem for the fractional fourier transform. IEEE Signal Processing Letters, 5(4):101–103, 1998.
  56. Deep stacked hierarchical multi-patch network for image deblurring. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  57. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2737–2746, 2020.
  58. Infwide: Image and feature space wiener deconvolution network for non-blind image deblurring in low-light conditions. IEEE Transactions on Image Processing, 32:1390–1402, 2023.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up