Papers
Topics
Authors
Recent
Search
2000 character limit reached

Uni-Removal: A Semi-Supervised Framework for Simultaneously Addressing Multiple Degradations in Real-World Images

Published 11 Jul 2023 in cs.CV and cs.AI | (2307.05075v1)

Abstract: Removing multiple degradations, such as haze, rain, and blur, from real-world images poses a challenging and illposed problem. Recently, unified models that can handle different degradations have been proposed and yield promising results. However, these approaches focus on synthetic images and experience a significant performance drop when applied to realworld images. In this paper, we introduce Uni-Removal, a twostage semi-supervised framework for addressing the removal of multiple degradations in real-world images using a unified model and parameters. In the knowledge transfer stage, Uni-Removal leverages a supervised multi-teacher and student architecture in the knowledge transfer stage to facilitate learning from pretrained teacher networks specialized in different degradation types. A multi-grained contrastive loss is introduced to enhance learning from feature and image spaces. In the domain adaptation stage, unsupervised fine-tuning is performed by incorporating an adversarial discriminator on real-world images. The integration of an extended multi-grained contrastive loss and generative adversarial loss enables the adaptation of the student network from synthetic to real-world domains. Extensive experiments on real-world degraded datasets demonstrate the effectiveness of our proposed method. We compare our Uni-Removal framework with state-of-the-art supervised and unsupervised methods, showcasing its promising results in real-world image dehazing, deraining, and deblurring simultaneously.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. W.-T. Chen, Z.-K. Huang, C.-C. Tsai, H.-H. Yang, J.-J. Ding, and S.-Y. Kuo, “Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 653–17 662.
  2. R. T. Tan, “Visibility in bad weather from a single image,” in 2008 IEEE conference on computer vision and pattern recognition.   IEEE, 2008, pp. 1–8.
  3. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 12, pp. 2341–2353, 2010.
  4. R. Fattal, “Dehazing using color-lines,” ACM transactions on graphics (TOG), vol. 34, no. 1, pp. 1–14, 2014.
  5. Z. Hu, S. Cho, J. Wang, and M.-H. Yang, “Deblurring low-light images with light streaks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3382–3389.
  6. W. Dong, L. Zhang, G. Shi, and X. Wu, “Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization,” IEEE Transactions on image processing, vol. 20, no. 7, pp. 1838–1857, 2011.
  7. B.-H. Chen and S.-C. Huang, “An advanced visibility restoration algorithm for single hazy images,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 11, no. 4, pp. 1–21, 2015.
  8. X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, “Clearing the skies: A deep network architecture for single-image rain removal,” IEEE Transactions on Image Processing, vol. 26, no. 6, pp. 2944–2956, 2017.
  9. T. Hyun Kim, B. Ahn, and K. Mu Lee, “Dynamic scene deblurring,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 3160–3167.
  10. Y. Wang, Y. Cao, J. Zhang, F. Wu, and Z.-J. Zha, “Leveraging deep statistics for underwater image enhancement,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 17, no. 3s, pp. 1–20, 2021.
  11. S. Jonna, M. Medhi, and R. R. Sahay, “Distill-dbdgan: Knowledge distillation and adversarial learning framework for defocus blur detection,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 2s, pp. 1–26, 2023.
  12. W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3253–3261.
  13. X.-S. Zhang, Y.-B. Yu, K.-F. Yang, and Y.-J. Li, “A fish retina-inspired single image dehazing method,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 4, pp. 1875–1888, 2021.
  14. X. Song, D. Zhou, W. Li, H. Ding, Y. Dai, and L. Zhang, “Wsamf-net: Wavelet spatial attention based multi-stream feedback network for single image dehazing,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  15. Y. Liu, Z. Yan, J. Tan, and Y. Li, “Multi-purpose oriented single nighttime image haze removal based on unified variational retinex model,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  16. Z. Hao, S. Gai, and P. Li, “Multi-scale self-calibrated dual-attention lightweight residual dense deraining network based on monogenic wavelets,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  17. L. Cai, Y. Fu, W. Huo, Y. Xiang, T. Zhu, Y. Zhang, H. Zeng, and D. Zeng, “Multi-scale attentive image de-raining networks via neural architecture search,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
  18. H. Li, J. Li, D. Zhao, and L. Xu, “DehazeFlow: Multi-scale conditional flow network for single image dehazing,” in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 2577–2585.
  19. P. Sharma, I. Bisht, and A. Sur, “Wavelength-based attributed deep neural network for underwater image restoration,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 1, pp. 1–23, 2023.
  20. K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep cnn denoiser prior for image restoration,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3929–3938.
  21. Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” arXiv preprint arXiv:1903.10082, 2019.
  22. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2480–2495, 2020.
  23. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 821–14 831.
  24. J. Xie, G. Hou, G. Wang, and Z. Pan, “A variational framework for underwater image dehazing and deblurring,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 6, pp. 3514–3526, 2021.
  25. Y. Mao, Z. Wan, Y. Dai, and X. Yu, “Deep idempotent network for efficient single image blind deblurring,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 1, pp. 172–185, 2022.
  26. R. Li, R. T. Tan, and L.-F. Cheong, “All in one bad weather removal using architectural search,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3175–3185.
  27. J. M. J. Valanarasu, R. Yasarla, and V. M. Patel, “Transweather: Transformer-based restoration of images degraded by adverse weather conditions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2353–2363.
  28. T. Wang, X. Yang, K. Xu, S. Chen, Q. Zhang, and R. W. Lau, “Spatial attentive single-image deraining with a high quality real rain dataset,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12 270–12 279.
  29. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on image processing, vol. 21, no. 12, pp. 4695–4708, 2012.
  30. N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in Twenty First National Conference on Communications (NCC).   IEEE, 2015, pp. 1–6.
  31. W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in European conference on computer vision.   Springer, 2016, pp. 154–169.
  32. B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “Dehazenet: An end-to-end system for single image haze removal,” IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187–5198, 2016.
  33. H. Zhang and V. M. Patel, “Densely connected pyramid dehazing network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3194–3203.
  34. L. Li, J. Pan, W.-S. Lai, C. Gao, N. Sang, and M.-H. Yang, “Dynamic scene deblurring by depth guided model,” IEEE Transactions on Image Processing, vol. 29, pp. 5273–5288, 2020.
  35. M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in International conference on machine learning.   PMLR, 2015, pp. 97–105.
  36. K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, “Unsupervised pixel-level domain adaptation with generative adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3722–3731.
  37. Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7472–7481.
  38. O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 8878–8887.
  39. Y. Shao, L. Li, W. Ren, C. Gao, and N. Sang, “Domain adaptation for image dehazing,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2808–2817.
  40. Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, and M. Wang, “Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking,” IEEE Transactions on Image Processing, vol. 30, pp. 4788–4801, 2021.
  41. Z. Chen, Y. Wang, Y. Yang, and D. Liu, “Psd: Principled synthetic-to-real dehazing guided by physical priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7180–7189.
  42. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for real image restoration and enhancement,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16.   Springer, 2020, pp. 492–511.
  43. J. Pan, S. Liu, D. Sun, J. Zhang, Y. Liu, J. Ren, Z. Li, J. Tang, H. Lu, Y.-W. Tai et al., “Learning dual convolutional neural networks for low-level vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3070–3079.
  44. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
  45. R. Adriana, B. Nicolas, K. S. Ebrahimi, C. Antoine, G. Carlo, and B. Yoshua, “Fitnets: Hints for thin deep nets,” Proc. ICLR, vol. 2, 2015.
  46. T. Wang, L. Yuan, X. Zhang, and J. Feng, “Distilling object detectors with fine-grained feature imitation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4933–4942.
  47. S. Ge, S. Zhao, C. Li, and J. Li, “Low-resolution face recognition in the wild via selective knowledge distillation,” IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 2051–2062, 2018.
  48. Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, “Structured knowledge distillation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2604–2613.
  49. M. Hong, Y. Xie, C. Li, and Y. Qu, “Distilling image dehazing with heterogeneous task imitation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3462–3471.
  50. M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics.   JMLR Workshop and Conference Proceedings, 2010, pp. 297–304.
  51. A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv preprint arXiv:1703.07737, 2017.
  52. K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” Advances in neural information processing systems, vol. 29, 2016.
  53. A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  54. T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, “Contrastive learning for unpaired image-to-image translation,” in European Conference on Computer Vision.   Springer, 2020, pp. 319–345.
  55. Y. Ye, C. Yu, Y. Chang, L. Zhu, X.-L. Zhao, L. Yan, and Y. Tian, “Unsupervised deraining: Where contrastive learning meets self-similarity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5821–5830.
  56. X. Chen, J. Pan, K. Jiang, Y. Li, Y. Huang, C. Kong, L. Dai, and Z. Fan, “Unpaired deep image deraining using dual contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2017–2026.
  57. H. Wu, Y. Qu, S. Lin, J. Zhou, R. Qiao, Z. Zhang, Y. Xie, and L. Ma, “Contrastive learning for compact single image dehazing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 551–10 560.
  58. X. Chen, Z.-C. Fan, Z. Zheng, Y. Li, Y. Huang, L. Dai, C. Kong, and P. Li, “Unpaired deep image dehazing using contrastive disentanglement learning,” 2022.
  59. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
  60. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  61. B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single-image dehazing and beyond,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 492–505, 2018.
  62. X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3855–3863.
  63. S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3883–3891.
  64. J. Rim, H. Lee, J. Won, and S. Cho, “Real-world blur dataset for learning and benchmarking deblurring algorithms,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16.   Springer, 2020, pp. 184–201.
  65. S.-W. Ji, J. Lee, S.-W. Kim, J.-P. Hong, S.-J. Baek, S.-W. Jung, and S.-J. Ko, “Xydeblur: divide and conquer for single image deblurring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 421–17 430.

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.