Uni-Removal: A Semi-Supervised Framework for Simultaneously Addressing Multiple Degradations in Real-World Images
Abstract: Removing multiple degradations, such as haze, rain, and blur, from real-world images poses a challenging and illposed problem. Recently, unified models that can handle different degradations have been proposed and yield promising results. However, these approaches focus on synthetic images and experience a significant performance drop when applied to realworld images. In this paper, we introduce Uni-Removal, a twostage semi-supervised framework for addressing the removal of multiple degradations in real-world images using a unified model and parameters. In the knowledge transfer stage, Uni-Removal leverages a supervised multi-teacher and student architecture in the knowledge transfer stage to facilitate learning from pretrained teacher networks specialized in different degradation types. A multi-grained contrastive loss is introduced to enhance learning from feature and image spaces. In the domain adaptation stage, unsupervised fine-tuning is performed by incorporating an adversarial discriminator on real-world images. The integration of an extended multi-grained contrastive loss and generative adversarial loss enables the adaptation of the student network from synthetic to real-world domains. Extensive experiments on real-world degraded datasets demonstrate the effectiveness of our proposed method. We compare our Uni-Removal framework with state-of-the-art supervised and unsupervised methods, showcasing its promising results in real-world image dehazing, deraining, and deblurring simultaneously.
- W.-T. Chen, Z.-K. Huang, C.-C. Tsai, H.-H. Yang, J.-J. Ding, and S.-Y. Kuo, “Learning multiple adverse weather removal via two-stage knowledge learning and multi-contrastive regularization: Toward a unified model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 653–17 662.
- R. T. Tan, “Visibility in bad weather from a single image,” in 2008 IEEE conference on computer vision and pattern recognition. IEEE, 2008, pp. 1–8.
- K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE transactions on pattern analysis and machine intelligence, vol. 33, no. 12, pp. 2341–2353, 2010.
- R. Fattal, “Dehazing using color-lines,” ACM transactions on graphics (TOG), vol. 34, no. 1, pp. 1–14, 2014.
- Z. Hu, S. Cho, J. Wang, and M.-H. Yang, “Deblurring low-light images with light streaks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3382–3389.
- W. Dong, L. Zhang, G. Shi, and X. Wu, “Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization,” IEEE Transactions on image processing, vol. 20, no. 7, pp. 1838–1857, 2011.
- B.-H. Chen and S.-C. Huang, “An advanced visibility restoration algorithm for single hazy images,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 11, no. 4, pp. 1–21, 2015.
- X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, “Clearing the skies: A deep network architecture for single-image rain removal,” IEEE Transactions on Image Processing, vol. 26, no. 6, pp. 2944–2956, 2017.
- T. Hyun Kim, B. Ahn, and K. Mu Lee, “Dynamic scene deblurring,” in Proceedings of the IEEE international conference on computer vision, 2013, pp. 3160–3167.
- Y. Wang, Y. Cao, J. Zhang, F. Wu, and Z.-J. Zha, “Leveraging deep statistics for underwater image enhancement,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 17, no. 3s, pp. 1–20, 2021.
- S. Jonna, M. Medhi, and R. R. Sahay, “Distill-dbdgan: Knowledge distillation and adversarial learning framework for defocus blur detection,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 2s, pp. 1–26, 2023.
- W. Ren, L. Ma, J. Zhang, J. Pan, X. Cao, W. Liu, and M.-H. Yang, “Gated fusion network for single image dehazing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3253–3261.
- X.-S. Zhang, Y.-B. Yu, K.-F. Yang, and Y.-J. Li, “A fish retina-inspired single image dehazing method,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 4, pp. 1875–1888, 2021.
- X. Song, D. Zhou, W. Li, H. Ding, Y. Dai, and L. Zhang, “Wsamf-net: Wavelet spatial attention based multi-stream feedback network for single image dehazing,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
- Y. Liu, Z. Yan, J. Tan, and Y. Li, “Multi-purpose oriented single nighttime image haze removal based on unified variational retinex model,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
- Z. Hao, S. Gai, and P. Li, “Multi-scale self-calibrated dual-attention lightweight residual dense deraining network based on monogenic wavelets,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
- L. Cai, Y. Fu, W. Huo, Y. Xiang, T. Zhu, Y. Zhang, H. Zeng, and D. Zeng, “Multi-scale attentive image de-raining networks via neural architecture search,” IEEE Transactions on Circuits and Systems for Video Technology, 2022.
- H. Li, J. Li, D. Zhao, and L. Xu, “DehazeFlow: Multi-scale conditional flow network for single image dehazing,” in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 2577–2585.
- P. Sharma, I. Bisht, and A. Sur, “Wavelength-based attributed deep neural network for underwater image restoration,” ACM Transactions on Multimedia Computing, Communications and Applications, vol. 19, no. 1, pp. 1–23, 2023.
- K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep cnn denoiser prior for image restoration,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3929–3938.
- Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” arXiv preprint arXiv:1903.10082, 2019.
- Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2480–2495, 2020.
- S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 821–14 831.
- J. Xie, G. Hou, G. Wang, and Z. Pan, “A variational framework for underwater image dehazing and deblurring,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 6, pp. 3514–3526, 2021.
- Y. Mao, Z. Wan, Y. Dai, and X. Yu, “Deep idempotent network for efficient single image blind deblurring,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 1, pp. 172–185, 2022.
- R. Li, R. T. Tan, and L.-F. Cheong, “All in one bad weather removal using architectural search,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3175–3185.
- J. M. J. Valanarasu, R. Yasarla, and V. M. Patel, “Transweather: Transformer-based restoration of images degraded by adverse weather conditions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2353–2363.
- T. Wang, X. Yang, K. Xu, S. Chen, Q. Zhang, and R. W. Lau, “Spatial attentive single-image deraining with a high quality real rain dataset,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12 270–12 279.
- A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on image processing, vol. 21, no. 12, pp. 4695–4708, 2012.
- N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in Twenty First National Conference on Communications (NCC). IEEE, 2015, pp. 1–6.
- W. Ren, S. Liu, H. Zhang, J. Pan, X. Cao, and M.-H. Yang, “Single image dehazing via multi-scale convolutional neural networks,” in European conference on computer vision. Springer, 2016, pp. 154–169.
- B. Cai, X. Xu, K. Jia, C. Qing, and D. Tao, “Dehazenet: An end-to-end system for single image haze removal,” IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187–5198, 2016.
- H. Zhang and V. M. Patel, “Densely connected pyramid dehazing network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3194–3203.
- L. Li, J. Pan, W.-S. Lai, C. Gao, N. Sang, and M.-H. Yang, “Dynamic scene deblurring by depth guided model,” IEEE Transactions on Image Processing, vol. 29, pp. 5273–5288, 2020.
- M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in International conference on machine learning. PMLR, 2015, pp. 97–105.
- K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, “Unsupervised pixel-level domain adaptation with generative adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3722–3731.
- Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, “Learning to adapt structured output space for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7472–7481.
- O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang, “Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 8878–8887.
- Y. Shao, L. Li, W. Ren, C. Gao, and N. Sang, “Domain adaptation for image dehazing,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2808–2817.
- Y. Wei, Z. Zhang, Y. Wang, M. Xu, Y. Yang, S. Yan, and M. Wang, “Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking,” IEEE Transactions on Image Processing, vol. 30, pp. 4788–4801, 2021.
- Z. Chen, Y. Wang, Y. Yang, and D. Liu, “Psd: Principled synthetic-to-real dehazing guided by physical priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 7180–7189.
- S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for real image restoration and enhancement,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16. Springer, 2020, pp. 492–511.
- J. Pan, S. Liu, D. Sun, J. Zhang, Y. Liu, J. Ren, Z. Li, J. Tang, H. Lu, Y.-W. Tai et al., “Learning dual convolutional neural networks for low-level vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3070–3079.
- G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
- R. Adriana, B. Nicolas, K. S. Ebrahimi, C. Antoine, G. Carlo, and B. Yoshua, “Fitnets: Hints for thin deep nets,” Proc. ICLR, vol. 2, 2015.
- T. Wang, L. Yuan, X. Zhang, and J. Feng, “Distilling object detectors with fine-grained feature imitation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4933–4942.
- S. Ge, S. Zhao, C. Li, and J. Li, “Low-resolution face recognition in the wild via selective knowledge distillation,” IEEE Transactions on Image Processing, vol. 28, no. 4, pp. 2051–2062, 2018.
- Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, “Structured knowledge distillation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2604–2613.
- M. Hong, Y. Xie, C. Li, and Y. Qu, “Distilling image dehazing with heterogeneous task imitation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3462–3471.
- M. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010, pp. 297–304.
- A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv preprint arXiv:1703.07737, 2017.
- K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” Advances in neural information processing systems, vol. 29, 2016.
- A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
- T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, “Contrastive learning for unpaired image-to-image translation,” in European Conference on Computer Vision. Springer, 2020, pp. 319–345.
- Y. Ye, C. Yu, Y. Chang, L. Zhu, X.-L. Zhao, L. Yan, and Y. Tian, “Unsupervised deraining: Where contrastive learning meets self-similarity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5821–5830.
- X. Chen, J. Pan, K. Jiang, Y. Li, Y. Huang, C. Kong, L. Dai, and Z. Fan, “Unpaired deep image deraining using dual contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2017–2026.
- H. Wu, Y. Qu, S. Lin, J. Zhou, R. Qiao, Z. Zhang, Y. Xie, and L. Ma, “Contrastive learning for compact single image dehazing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 551–10 560.
- X. Chen, Z.-C. Fan, Z. Zheng, Y. Li, Y. Huang, L. Dai, C. Kong, and P. Li, “Unpaired deep image dehazing using contrastive disentanglement learning,” 2022.
- J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2223–2232.
- K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
- B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single-image dehazing and beyond,” IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 492–505, 2018.
- X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3855–3863.
- S. Nah, T. Hyun Kim, and K. Mu Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3883–3891.
- J. Rim, H. Lee, J. Won, and S. Cho, “Real-world blur dataset for learning and benchmarking deblurring algorithms,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16. Springer, 2020, pp. 184–201.
- S.-W. Ji, J. Lee, S.-W. Kim, J.-P. Hong, S.-J. Baek, S.-W. Jung, and S.-J. Ko, “Xydeblur: divide and conquer for single image deblurring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 17 421–17 430.
Paper Prompts
Sign up for free to create and run prompts on this paper.