Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NiteDR: Nighttime Image De-Raining with Cross-View Sensor Cooperative Learning for Dynamic Driving Scenes (2402.18172v2)

Published 28 Feb 2024 in cs.CV

Abstract: In real-world environments, outdoor imaging systems are often affected by disturbances such as rain degradation. Especially, in nighttime driving scenes, insufficient and uneven lighting shrouds the scenes in darkness, resulting degradation of both the image quality and visibility. Particularly, in the field of autonomous driving, the visual perception ability of RGB sensors experiences a sharp decline in such harsh scenarios. Additionally, driving assistance systems suffer from reduced capabilities in capturing and discerning the surrounding environment, posing a threat to driving safety. Single-view information captured by single-modal sensors cannot comprehensively depict the entire scene. To address these challenges, we developed an image de-raining framework tailored for rainy nighttime driving scenes. It aims to remove rain artifacts, enrich scene representation, and restore useful information. Specifically, we introduce cooperative learning between visible and infrared images captured by different sensors. By cross-view fusion of these multi-source data, the scene within the images gains richer texture details and enhanced contrast. We constructed an information cleaning module called CleanNet as the first stage of our framework. Moreover, we designed an information fusion module called FusionNet as the second stage to fuse the clean visible images with infrared images. Using this stage-by-stage learning strategy, we obtain de-rained fusion images with higher quality and better visual perception. Extensive experiments demonstrate the effectiveness of our proposed Cross-View Cooperative Learning (CVCL) in adverse driving scenarios in low-light rainy environments. The proposed approach addresses the gap in the utilization of existing rain removal algorithms in specific low-light conditions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 17 683–17 693.
  2. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5728–5739.
  3. V. Frants, S. Agaian, and K. Panetta, “Qsam-net: Rain streak removal by quaternion neural network with self-attention module,” IEEE Transactions on Multimedia, 2023.
  4. X. Chen, J. Pan, J. Lu, Z. Fan, and H. Li, “Hybrid cnn-transformer feature fusion for single image deraining,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 1, 2023, pp. 378–386.
  5. H. Li, X.-J. Wu, and J. Kittler, “Rfn-nest: An end-to-end residual fusion network for infrared and visible images,” Information Fusion, vol. 73, pp. 72–86, 2021.
  6. L. Tang, J. Yuan, and J. Ma, “Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network,” Information Fusion, vol. 82, pp. 28–42, 2022.
  7. W. Zhao, S. Xie, F. Zhao, Y. He, and H. Lu, “Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 955–13 965.
  8. H. Zhang and V. M. Patel, “Convolutional sparse and low-rank coding-based rain streak removal,” in 2017 IEEE Winter conference on applications of computer vision (WACV).   IEEE, 2017, pp. 1259–1267.
  9. S. Gu, D. Meng, W. Zuo, and L. Zhang, “Joint convolutional analysis and synthesis sparse representation for single image layer separation,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1708–1716.
  10. L.-W. Kang, C.-W. Lin, and Y.-H. Fu, “Automatic single-image-based rain streaks removal via image decomposition,” IEEE transactions on image processing, vol. 21, no. 4, pp. 1742–1755, 2011.
  11. Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown, “Rain streak removal using layer priors,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2736–2744.
  12. Y. Luo, Y. Xu, and H. Ji, “Removing rain from a single image via discriminative sparse coding,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 3397–3405.
  13. Z. Jin, M. Z. Iqbal, D. Bobkov, W. Zou, X. Li, and E. Steinbach, “A flexible deep cnn framework for image restoration,” IEEE Transactions on Multimedia, vol. 22, no. 4, pp. 1055–1068, 2019.
  14. X. Chen, J. Pan, K. Jiang, Y. Li, Y. Huang, C. Kong, L. Dai, and Z. Fan, “Unpaired deep image deraining using dual contrastive learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2017–2026.
  15. A. Kulkarni, P. W. Patil, S. Murala, and S. Gupta, “Unified multi-weather visibility restoration,” IEEE Transactions on Multimedia, 2022.
  16. K. Jiang, Z. Wang, P. Yi, C. Chen, Z. Han, T. Lu, B. Huang, and J. Jiang, “Decomposition makes better rain removal: An improved attention-guided deraining network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3981–3995, 2020.
  17. Y. Wang, D. Gong, J. Yang, Q. Shi, D. Xie, B. Zeng et al., “Deep single image deraining via modeling haze-like effect,” IEEE Transactions on Multimedia, vol. 23, pp. 2481–2492, 2020.
  18. K. Jiang, Z. Wang, P. Yi, C. Chen, Z. Wang, X. Wang, J. Jiang, and C.-W. Lin, “Rain-free and residue hand-in-hand: A progressive coupled network for real-time image deraining,” IEEE Transactions on Image Processing, vol. 30, pp. 7404–7418, 2021.
  19. Y. Yang, J. Guan, S. Huang, W. Wan, Y. Xu, and J. Liu, “End-to-end rain removal network based on progressive residual detail supplement,” IEEE Transactions on Multimedia, vol. 24, pp. 1622–1636, 2021.
  20. Q. Wang, K. Jiang, Z. Wang, W. Ren, J. Zhang, and C.-W. Lin, “Multi-scale fusion and decomposition network for single image deraining,” IEEE Transactions on Image Processing, vol. 33, pp. 191–204, 2023.
  21. X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 3855–3863.
  22. X. Lin, L. Ma, B. Sheng, Z.-J. Wang, and W. Chen, “Utilizing two-phase processing with fbls for single image deraining,” IEEE Transactions on Multimedia, vol. 23, pp. 664–676, 2020.
  23. Y. Que, S. Li, and H. J. Lee, “Attentive composite residual network for robust rain removal from single images,” IEEE Transactions on Multimedia, vol. 23, pp. 3059–3072, 2020.
  24. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  25. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  26. H. Chen, D. Jiang, and H. Sahli, “Transformer encoder with multi-modal multi-head attention for continuous affect recognition,” IEEE Transactions on Multimedia, vol. 23, pp. 4171–4183, 2020.
  27. W. Zhou, Y. Cai, L. Zhang, W. Yan, and L. Yu, “Utlnet: Uncertainty-aware transformer localization network for rgb-depth mirror segmentation,” IEEE Transactions on Multimedia, 2023.
  28. Y. Liang, S. Anwar, and Y. Liu, “Drt: A lightweight single image deraining recursive transformer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 589–598.
  29. Q. Qin, J. Yan, Q. Wang, X. Wang, M. Li, and Y. Wang, “Etdnet: An efficient transformer deraining model,” IEEE Access, vol. 9, pp. 119 881–119 893, 2021.
  30. J. Xiao, X. Fu, A. Liu, F. Wu, and Z.-J. Zha, “Image de-raining transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  31. H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 299–12 310.
  32. K. Jiang, Z. Wang, C. Chen, Z. Wang, L. Cui, and C.-W. Lin, “Magic elf: Image deraining meets association learning and transformer,” arXiv preprint arXiv:2207.10455, 2022.
  33. S. Chen, T. Ye, J. Bai, E. Chen, J. Shi, and L. Zhu, “Sparse sampling transformer with uncertainty-driven ranking for unified removal of raindrops and rain streaks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 13 106–13 117.
  34. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, “Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2017, pp. 5108–5115.
  35. Y. Lu, Y. Wu, B. Liu, T. Zhang, B. Li, Q. Chu, and N. Yu, “Cross-modality person re-identification with shared-specific feature transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13 379–13 389.
  36. Z. Zhou, B. Wang, S. Li, and M. Dong, “Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters,” Information Fusion, vol. 30, pp. 15–26, 2016.
  37. H. Li, X. Qi, and W. Xie, “Fast infrared and visible image fusion with structural decomposition,” Knowledge-Based Systems, vol. 204, p. 106182, 2020.
  38. J. Ma and Y. Zhou, “Infrared and visible image fusion via gradientlet filter,” Computer Vision and Image Understanding, vol. 197, p. 103016, 2020.
  39. Y. Liu, J. Jin, Q. Wang, Y. Shen, and X. Dong, “Region level based multi-focus image fusion using quaternion wavelet and normalized cut,” Signal Processing, vol. 97, pp. 9–30, 2014.
  40. V. Bhateja, H. Patel, A. Krishn, A. Sahu, and A. Lay-Ekuakille, “Multimodal medical image sensor fusion framework using cascade of wavelet and contourlet transform domains,” IEEE Sensors Journal, vol. 15, no. 12, pp. 6783–6790, 2015.
  41. Y. Yang, Y. Que, S. Huang, and P. Lin, “Multimodal sensor medical image fusion based on type-2 fuzzy logic in nsct domain,” IEEE Sensors Journal, vol. 16, no. 10, pp. 3735–3745, 2016.
  42. C.-I. Chen, “Fusion of pet and mr brain images based on ihs and log-gabor transforms,” IEEE Sensors Journal, vol. 17, no. 21, pp. 6995–7010, 2017.
  43. Q. Jiang, X. Jin, J. Hou, S.-J. Lee, and S. Yao, “Multi-sensor image fusion based on interval type-2 fuzzy sets and regional features in nonsubsampled shearlet transform domain,” IEEE Sensors Journal, vol. 18, no. 6, pp. 2494–2505, 2018.
  44. Y. Liu, X. Chen, R. K. Ward, and Z. J. Wang, “Image fusion with convolutional sparse representation,” IEEE signal processing letters, vol. 23, no. 12, pp. 1882–1886, 2016.
  45. M. Wu, Y. Ma, F. Fan, X. Mei, and J. Huang, “Infrared and visible image fusion via joint convolutional sparse representation,” JOSA A, vol. 37, no. 7, pp. 1105–1115, 2020.
  46. N. Cvejic, D. Bull, and N. Canagarajah, “Region-based multimodal image fusion using ica bases,” IEEE Sensors Journal, vol. 7, no. 5, pp. 743–751, 2007.
  47. Z. Fu, X. Wang, J. Xu, N. Zhou, and Y. Zhao, “Infrared and visible images fusion based on rpca and nsct,” Infrared Physics & Technology, vol. 77, pp. 114–123, 2016.
  48. J. Ma, C. Chen, C. Li, and J. Huang, “Infrared and visible image fusion via gradient transfer and total variation minimization,” Information Fusion, vol. 31, pp. 100–109, 2016.
  49. P. Ganasala and A. Prasad, “Contrast enhanced multi sensor image fusion based on guided image filter and nsst,” IEEE Sensors Journal, vol. 20, no. 2, pp. 939–946, 2019.
  50. Z. Zhao, S. Xu, J. Zhang, C. Liang, C. Zhang, and J. Liu, “Efficient and model-based infrared and visible image fusion via algorithm unrolling,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 3, pp. 1186–1196, 2021.
  51. H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, “U2fusion: A unified unsupervised image fusion network,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502–518, 2020.
  52. J. Ma, L. Tang, F. Fan, J. Huang, X. Mei, and Y. Ma, “Swinfusion: Cross-domain long-range learning for general image fusion via swin transformer,” IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200–1217, 2022.
  53. J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang, “Fusiongan: A generative adversarial network for infrared and visible image fusion,” Information fusion, vol. 48, pp. 11–26, 2019.
  54. J. Ma, H. Xu, J. Jiang, X. Mei, and X.-P. Zhang, “Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion,” IEEE Transactions on Image Processing, vol. 29, pp. 4980–4995, 2020.
  55. J. Li, H. Huo, C. Li, R. Wang, and Q. Feng, “Attentionfgan: Infrared and visible image fusion using attention-based generative adversarial networks,” IEEE Transactions on Multimedia, vol. 23, pp. 1383–1396, 2020.
  56. L. Tang, J. Yuan, H. Zhang, X. Jiang, and J. Ma, “Piafusion: A progressive infrared and visible image fusion network based on illumination aware,” Information Fusion, vol. 83, pp. 79–92, 2022.
  57. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  58. C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zero-reference deep curve estimation for low-light image enhancement,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 1780–1789.
  59. J. Choi, D. H. Kim, S. Lee, S. H. Lee, and B. C. Song, “Synthesized rain images for deraining algorithms,” Neurocomputing, vol. 492, pp. 421–439, 2022.
  60. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 821–14 831.
  61. X. Chen, H. Li, M. Li, and J. Pan, “Learning a sparse transformer network for effective image deraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5896–5905.
  62. Y. Fu, X.-J. Wu, and T. Durrani, “Image fusion based on generative adversarial network consistent with perception,” Information Fusion, vol. 72, pp. 110–125, 2021.
  63. J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, “Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5802–5811.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - CidanShi/NiteDR-Nighttime-Image-De-raining (8 stars)