Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Mixed Hierarchy Network for Image Restoration (2302.09554v4)

Published 19 Feb 2023 in cs.CV

Abstract: Image restoration is a long-standing low-level vision problem, e.g., deblurring and deraining. In the process of image restoration, it is necessary to consider not only the spatial details and contextual information of restoration to ensure the quality, but also the system complexity. Although many methods have been able to guarantee the quality of image restoration, the system complexity of the state-of-the-art (SOTA) methods is increasing as well. Motivated by this, we present a mixed hierarchy network that can balance these competing goals. Our main proposal is a mixed hierarchy architecture, that progressively recovers contextual information and spatial details from degraded images while we design intra-blocks to reduce system complexity. Specifically, our model first learns the contextual information using encoder-decoder architectures, and then combines them with high-resolution branches that preserve spatial detail. In order to reduce the system complexity of this architecture for convenient analysis and comparison, we replace or remove the nonlinear activation function with multiplication and use a simple network structure. In addition, we replace spatial convolution with global self-attention for the middle block of encoder-decoder. The resulting tightly interlinked hierarchy architecture, named as MHNet, delivers strong performance gains on several image restoration tasks, including image deraining, and deblurring.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D Nonlinear Phenomena, vol. 60, no. 1-4, pp. 259–268, 1992.
  2. P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 7, pp. 629–639, 2002.
  3. S. Roth and M. J. Black, “Fields of experts: A framework for learning image priors,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005.
  4. W. Dong, L. Zhang, G. Shi, and X. Wu, “Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization,” IEEE Transactions on Image Processing, vol. 20, no. 7, pp. 1838–1857, 2011.
  5. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011.
  6. T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 11 065–11 074.
  7. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” pp. 1–1, 2020.
  8. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in CVPR, 2021.
  9. ——, “Learning enriched features for fast image restoration and enhancement,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
  10. L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” arXiv preprint arXiv:2204.04676, 2022.
  11. X. Chu, L. Chen, and W. Yu, “Nafssr: Stereo image super-resolution using nafnet,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2022, pp. 1239–1248.
  12. Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 17 683–17 693.
  13. L. Chen, X. Lu, J. Zhang, X. Chu, and C. Chen, “Hinet: Half instance normalization network for image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2021, pp. 182–192.
  14. D. Ren, W. Zuo, Q. Hu, P. F. Zhu, and D. Meng, “Progressive image deraining networks: A better and simpler baseline,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3932–3941, 2019.
  15. X. Li, J. Wu, Z. Lin, H. Liu, and H. Zha, “Recurrent squeeze-and-excitation context aggregation net for single image deraining,” in European Conference on Computer Vision, 2018.
  16. J. Pan, S. Liu, D. Sun, J. Zhang, Y. Liu, J. Ren, Z. Li, J. Tang, H. Lu, and Y. W. a. Tai, “Learning dual convolutional neural networks for low-level vision,” in CVPR, 2018.
  17. J. Pan, D. Sun, J. Zhang, J. Tang, J. Yang, Y. W. Tai, and M. H. Yang, “Dual convolutional neural networks for low-level vision,” International Journal of Computer Vision, 2022.
  18. V. Singh, K. Ramnath, and A. Mittal, “Refining high-frequencies for sharper super-resolution and deblurring,” Computer Vision and Image Understanding, vol. 199, no. C, p. 103034, 2020.
  19. D. Chen and M. E. Davies, “Deep decomposition learning for inverse imaging problems,” in Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  20. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 286–301.
  21. S. Anwar, “Real image denoising with feature attention,” ICCV, 2019.
  22. J. Zhang, Y. Zhang, J. Gu, Y. Zhang, L. Kong, and X. Yuan, “Accurate image restoration with attention retractable transformer,” in ICLR, 2023.
  23. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in CVPR, 2022.
  24. F.-J. Tsai, Y.-T. Peng, Y.-Y. Lin, C.-C. Tsai, and C.-W. Lin, “Stripformer: Strip transformer for fast image deblurring,” in ECCV, 2022.
  25. T. F. Chan and C.-K. Wong, “Total variation blind deconvolution,” IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, vol. 7 3, pp. 370–5, 1998.
  26. L. Yu, X. Yong, and J. Hui, “Removing rain from a single image via discriminative sparse coding,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
  27. M. Aharon, M. Elad, and A. Bruckstein, “K-svd: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE TRANSACTIONS ON SIGNAL PROCESSING, 2006.
  28. J. Mairal, M. Elad, and G. Sapiro, “Sparse representation for color image restoration,” IEEE Transactions on Image Processing, vol. 17, no. 1, pp. 53–69, 2007.
  29. A. Buades, B. Coll, and J. M. Morel, “A non-local algorithm for image denoising,” in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, 2005.
  30. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” in IEEE, 2007, pp. 2080–2095.
  31. S. Qi, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image,” Acm Transactions on Graphics, vol. 27, no. 3, pp. 1–10, 2008.
  32. X. Li, S. Zheng, and J. Jia, “Unnatural l0 sparse representation for natural image deblurring,” in IEEE Conference on Computer Vision and Pattern Recognition, 2013.
  33. X. Chu, L. Chen, , C. Chen, and X. Lu, “Improving image restoration by revisiting global information aggregation,” arXiv preprint arXiv:2112.04491, 2021.
  34. X. J. Mao, C. Shen, and Y. B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” in NIPS, 2016.
  35. I. D. Mastan and S. Raman, “Multi-level encoder-decoder architectures for image restoration,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1728–1737, 2019.
  36. J. Su, B. Xu, and H. Yin, “A survey of deep learning approaches to image restoration,” Neurocomputing, vol. 487, pp. 46–65, 2022.
  37. T. Zeng, H. So, and E. Lam, “Redcap: residual encoder-decoder capsule network for holographic image reconstruction,” Optics Express, 2020.
  38. S. Cheng, Y. Wang, H. Huang, D. Liu, H. Fan, and S. Liu, “NBNet: Noise Basis Learning for Image Denoising with Subspace Projection,” CVPR, Dec. 2021.
  39. X. Tao, H. Gao, Y. Wang, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” CVPR, 2018.
  40. X. Fu, B. Liang, Y. Huang, X. Ding, and J. Paisley, “Lightweight pyramid networks for image deraining,” IEEE Transactions on Neural Networks and Learning Systems, 2018.
  41. H. Zhang, L. Zhang, Y. Dai, H. Li, and P. Koniusz, “Event-guided multi-patch network with self-supervision for non-uniform motion deblurring,” International Journal of Computer Vision, pp. 1–18, 2022.
  42. H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  43. H. Zhu, P. Xi, V. Chandrasekhar, L. Li, and J. H. Lim, “Dehazegan: When image dehazing meets differential programming,” in Twenty-Seventh International Joint Conference on Artificial Intelligence IJCAI-18, 2018.
  44. T. Guo, X. Li, V. Cherukuri, and V. Monga, “Dense scene information estimation network for dehazing,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019.
  45. A. Yang, H. Wang, Z. Ji, Y. Pang, and L. Shao, “Dual-path in dual-path network for single image dehazing,” in Twenty-Eighth International Joint Conference on Artificial Intelligence IJCAI-19, 2019.
  46. L. I. Siyuan, W. Ren, J. Zhang, J. Yu, and X. Guo, “Fast single image rain removal via a deep decomposition-composition network,” 2018.
  47. C. Tian, Y. Xu, W. Zuo, B. Du, C.-W. Lin, and D. Zhang, “Designing and training of a dual cnn for image denoising,” Knowledge-Based Systems, vol. 226, p. 106949, 2021.
  48. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” arXiv, 2017.
  49. M. V. Conde and K. Turgutlu, “Clip-art: Contrastive pre-training for fine-grained art classification,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2021, pp. 3956–3960.
  50. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021.
  51. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in ICCV 2021, October 2021.
  52. X. Chen, C.-J. Hsieh, and B. Gong, “When vision transformers outperform resnets without pretraining or strong data augmentations,” arXiv preprint arXiv:2106.01548, 2021.
  53. M. V. Conde, U.-J. Choi, M. Burchi, and R. Timofte, “Swin2SR: Swinv2 transformer for compressed image super-resolution and restoration,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2022.
  54. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” arXiv preprint arXiv:2108.10257, 2021.
  55. X. Chen, H. Li, M. Li, and J. Pan, “Learning a sparse transformer network for effective image deraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 5896–5905.
  56. L. Kong, J. Dong, J. Ge, M. Li, and J. Pan, “Efficient frequency domain-based transformers for high-quality image deblurring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5886–5895.
  57. J. Xiao, X. Fu, A. Liu, F. Wu, and Z.-J. Zha, “Image de-raining transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 11, pp. 12 978–12 995, 2023.
  58. T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, and J. T. Barron, “Unprocessing images for learned raw denoising,” 2018.
  59. C. Chen, C. Qifeng, X. Jia, and K. Vladlen, “Learning to see in the dark,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  60. O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8877–8886, 2019.
  61. X. Li, W. Wang, X. Hu, and J. Yang, “Selective kernel networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 510–519.
  62. H. Zhang, V. A. Sindagi, and V. M. Patel, “Image de-raining using a conditional generative adversarial network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, pp. 3943–3956, 2017.
  63. K. Jiang, Z. Wang, P. Yi, C. Chen, B. Huang, Y. Luo, J. Ma, and J. Jiang, “Multi-scale progressive fusion network for single image deraining,” CVPR, 2020.
  64. W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan, “Deep joint rain detection and removal from a single image,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685–1694, 2016.
  65. X. Fu, J. Huang, X. Ding, Y. Liao, and J. W. Paisley, “Clearing the skies: A deep network architecture for single-image rain removal,” IEEE Transactions on Image Processing, vol. 26, pp. 2944–2956, 2016.
  66. W. Wei, D. Meng, Q. Zhao, and Z. Xu, “Semi-supervised cnn for single image rain removal,” ArXiv, vol. abs/1807.11078, 2018.
  67. H. Zhang and V. M. Patel, “Density-aware single image de-raining using a multi-stream dense network,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 695–704, 2018.
  68. R. Yasarla and V. M. Patel, “Uncertainty guided multi-scale residual learning-using a cycle spinning cnn for single image de-raining,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8397–8406, 2019.
  69. K. Purohit, M. Suin, A. N. Rajagopalan, and V. N. Boddeti, “Spatially-adaptive image restoration using distortion-guided networks,” CoRR, vol. abs/2108.08617, 2021.
  70. X. Feng, H. Ji, W. Pei, J. Li, G. Lu, and D. Zhang, “U2-former: Nested u-shaped transformer for image restoration via multi-view contrastive learning,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2023.
  71. Z. Hao, S. Gai, and P. Li, “Multi-scale self-calibrated dual-attention lightweight residual dense deraining network based on monogenic wavelets,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 6, pp. 2642–2655, 2023.
  72. W. Wu, Y. Liu, and Z. Li, “Subband differentiated learning network for rain streak removal,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 9, pp. 4675–4688, 2023.
  73. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Computer Science, 2014.
  74. I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” 2016.
  75. S. Nah, T. H. Kim, and K. M. Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 257–265, 2016.
  76. Z. Shen, W. Wang, X. Lu, J. Shen, H. Ling, T. Xu, and L. Shao, “Human-aware motion deblurring,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5571–5580, 2019.
  77. Y. Zhang, Q. Li, M. Qi, D. Liu, J. Kong, and J. Wang, “Multi-scale frequency separation network for image deblurring,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 10, pp. 5525–5537, 2023.
  78. O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8183–8192, 2017.
  79. H. Gao, X. Tao, X. Shen, and J. Jia, “Dynamic scene deblurring with parameter selective sharing and nested skip connections,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3843–3851, 2019.
  80. K. Zhang, W. Luo, Y. Zhong, L. Ma, B. Stenger, W. Liu, and H. Li, “Deblurring by realistic blurring,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2734–2743, 2020.
  81. D. Park, D. U. Kang, J. Kim, and S. Y. Chun, “Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training,” in European Conference on Computer Vision, 2019.
  82. M. Suin, K. Purohit, and A. N. Rajagopalan, “Spatially-attentive patch-hierarchical network for adaptive motion deblurring,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3603–3612, 2020.
  83. S. J. Cho, S. W. Ji, J. P. Hong, S. W. Jung, and S. J. Ko, “Rethinking coarse-to-fine approach in single image deblurring,” in ICCV, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Hu Gao (15 papers)
  2. Depeng Dang (16 papers)
Citations (10)
View on Semantic Scholar