Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NIR-Assisted Image Denoising: A Selective Fusion Approach and A Real-World Benchmark Dataset (2404.08514v3)

Published 12 Apr 2024 in cs.CV

Abstract: Despite the significant progress in image denoising, it is still challenging to restore fine-scale details while removing noise, especially in extremely low-light environments. Leveraging near-infrared (NIR) images to assist visible RGB image denoising shows the potential to address this issue, becoming a promising technology. Nonetheless, existing works still struggle with taking advantage of NIR information effectively for real-world image denoising, due to the content inconsistency between NIR-RGB images and the scarcity of real-world paired datasets. To alleviate the problem, we propose an efficient Selective Fusion Module (SFM), which can be plug-and-played into the advanced denoising networks to merge the deep NIR-RGB features. Specifically, we sequentially perform the global and local modulation for NIR and RGB features, and then integrate the two modulated features. Furthermore, we present a Real-world NIR-Assisted Image Denoising (Real-NAID) dataset, which covers diverse scenarios as well as various noise levels. Extensive experiments on both synthetic and our real-world datasets demonstrate that the proposed method achieves better results than state-of-the-art ones. The dataset, codes, and pre-trained models will be publicly available at https://github.com/ronjonxu/NAID.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016.
  2. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in ICCV, 2021.
  3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017.
  4. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE transactions on image processing, vol. 26, no. 7, pp. 3142–3155, 2017.
  5. K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible solution for cnn-based image denoising,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4608–4622, 2018.
  6. A. Abdelhamed, M. Afifi, R. Timofte, and M. S. Brown, “Ntire 2020 challenge on real image denoising: Dataset, methods and results,” in CVPR Workshops, 2020, pp. 496–497.
  7. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5728–5739.
  8. Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 17 683–17 693.
  9. J. Li, Z. Zhang, X. Liu, C. Feng, X. Wang, L. Lei, and W. Zuo, “Spatially adaptive self-supervised learning for real-world image denoising,” in CVPR, 2023.
  10. J. Zhang, Y. Zhang, J. Gu, J. Dong, L. Kong, and X. Yang, “Xformer: Hybrid x-shaped transformer for image denoising,” arXiv preprint arXiv:2303.06440, 2023.
  11. B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, and R. Carroll, “Burst denoising with kernel prediction networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2502–2510.
  12. C. Godard, K. Matzen, and M. Uyttendaele, “Deep burst denoising,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 538–554.
  13. N. Pearl, T. Treibitz, and S. Korman, “Nan: Noise-aware nerfs for burst-denoising,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12 672–12 681.
  14. R. Wu, Z. Zhang, S. Zhang, H. Zhang, and W. Zuo, “Rbsr: Efficient and flexible recurrent network for burst super-resolution,” in Chinese Conference on Pattern Recognition and Computer Vision (PRCV).   Springer, 2023, pp. 65–78.
  15. Z. Zhang, R. Xu, M. Liu, Z. Yan, and W. Zuo, “Self-supervised image restoration with blurry and noisy pairs,” Advances in Neural Information Processing Systems, vol. 35, pp. 29 179–29 191, 2022.
  16. Z. Zhang, S. Zhang, R. Wu, Z. Yan, and W. Zuo, “Exposure bracketing is all you need for unifying image restoration and enhancement tasks,” arXiv preprint arXiv:2401.00766, 2024.
  17. F. Lv, Y. Zheng, Y. Li, and F. Lu, “An integrated enhancement solution for 24-hour colorful imaging,” in Proceedings of the AAAI conference on artificial intelligence, 2020, pp. 11 725–11 732.
  18. G. Wu, Y. Zheng, Z. Guo, Z. Cai, X. Shi, X. Ding, Y. Huang, Y. Guo, and R. Shibasaki, “Learn to recover visible color for video surveillance in a day,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16.   Springer, 2020, pp. 495–511.
  19. S. Jin, B. Yu, M. Jing, Y. Zhou, J. Liang, and R. Ji, “Darkvisionnet: Low-light imaging via rgb-nir fusion with deep inconsistency prior,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2022, pp. 1104–1112.
  20. R. Wan, B. Shi, W. Yang, B. Wen, L.-Y. Duan, and A. C. Kot, “Purifying low-light images via near-infrared enlightened image,” IEEE Transactions on Multimedia, 2022.
  21. C. Fredembach and S. Süsstrunk, “Colouring the near infrared,” in Proceedings of the IS&T;/SID 16th Color Imaging Conference, 2008, pp. 176–182.
  22. J. Xiong, J. Wang, W. Heidrich, and S. Nayar, “Seeing in extra darkness using a deep-red flash,” in CVPR, 2021, pp. 10 000–10 009.
  23. Z. Sheng, Z. Yu, X. Liu, S.-Y. Cao, Y. Liu, H.-L. Shen, and H. Zhang, “Structure aggregation for cross-spectral stereo image guided denoising,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 997–14 006.
  24. L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” in European Conference on Computer Vision.   Springer, 2022, pp. 17–33.
  25. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI.   Springer, 2015.
  26. H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” in CVPR, 2021.
  27. A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 2.   Ieee, 2005, pp. 60–65.
  28. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Transactions on image processing, vol. 16, no. 8, pp. 2080–2095, 2007.
  29. S. Gu, L. Zhang, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 2862–2869.
  30. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 821–14 831.
  31. D. Krishnan and R. Fergus, “Dark flash photography,” ACM Trans. Graph., vol. 28, no. 3, jul 2009. [Online]. Available: https://doi.org/10.1145/1531326.1531402
  32. X. Wang, F. Dai, Y. Ma, J. Guo, Q. Zhao, and Y. Zhang, “Near-infrared image guided neural networks for color image denoising,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2019, pp. 3807–3811.
  33. Y. Yang, J. Liu, S. Huang, W. Wan, W. Wen, and J. Guan, “Infrared and visible image fusion via texture conditional generative adversarial network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 12, pp. 4771–4783, 2021.
  34. Y. Cheng, R. Yang, Z. Zhang, J. Suo, and Q. Dai, “A mutually boosting dual sensor computational camera for high quality dark videography,” Information Fusion, vol. 93, pp. 429–440, 2023.
  35. Z. Zhao, H. Bai, J. Zhang, Y. Zhang, S. Xu, Z. Lin, R. Timofte, and L. Van Gool, “Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 5906–5916.
  36. X. Deng and P. L. Dragotti, “Deep convolutional neural network for multi-modal image restoration and fusion,” IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 10, pp. 3333–3348, 2020.
  37. S. Xu, J. Zhang, J. Wang, K. Sun, C. Zhang, J. Liu, and J. Hu, “A model-driven network for guided image denoising,” Information Fusion, vol. 85, pp. 60–71, 2022.
  38. M. Brown and S. Süsstrunk, “Multi-spectral sift for scene category recognition,” in CVPR 2011.   IEEE, 2011, pp. 177–184.
  39. T. Zhi, B. R. Pires, M. Hebert, and S. G. Narasimhan, “Deep material-aware cross-spectral stereo matching,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1916–1925.
  40. J. Wang, T. Xue, J. T. Barron, and J. Chen, “Stereoscopic dark flash for low-light photography,” in 2019 IEEE International Conference on Computational Photography (ICCP).   IEEE, 2019, pp. 1–10.
  41. J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
  42. K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV, 2015, pp. 1026–1034.
  43. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017.
  44. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  45. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
  46. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  47. Z. Sheng, X. Liu, S.-Y. Cao, H.-L. Shen, and H. Zhang, “Frequency-domain deep guided image denoising,” IEEE Transactions on Multimedia, 2022.
  48. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  49. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018, pp. 586–595.
  50. Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in ECCV, 2018, pp. 286–301.
  51. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in ICCV, 2021, pp. 10 012–10 022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets