Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stimulating Diffusion Model for Image Denoising via Adaptive Embedding and Ensembling (2307.03992v5)

Published 8 Jul 2023 in cs.CV

Abstract: Image denoising is a fundamental problem in computational photography, where achieving high perception with low distortion is highly demanding. Current methods either struggle with perceptual quality or suffer from significant distortion. Recently, the emerging diffusion model has achieved state-of-the-art performance in various tasks and demonstrates great potential for image denoising. However, stimulating diffusion models for image denoising is not straightforward and requires solving several critical problems. For one thing, the input inconsistency hinders the connection between diffusion models and image denoising. For another, the content inconsistency between the generated image and the desired denoised image introduces distortion. To tackle these problems, we present a novel strategy called the Diffusion Model for Image Denoising (DMID) by understanding and rethinking the diffusion model from a denoising perspective. Our DMID strategy includes an adaptive embedding method that embeds the noisy image into a pre-trained unconditional diffusion model and an adaptive ensembling method that reduces distortion in the denoised image. Our DMID strategy achieves state-of-the-art performance on both distortion-based and perception-based metrics, for both Gaussian and real-world image denoising.The code is available at https://github.com/Li-Tong-621/DMID.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (92)
  1. M. Aharon, M. Elad, and A. Bruckstein, “K-svd: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Transactions on Signal Processing, pp. 4311–4322, 2006.
  2. S. Gu, Z. Lei, W. Zuo, and X. Feng, “Weighted nuclear norm minimization with application to image denoising,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2862–2869, 2014.
  3. J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using scale mixtures of gaussians in the wavelet domain,” IEEE Transactions on Image processing (TIP), pp. 1338–1351, 2003.
  4. J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1833–1844, 2021.
  5. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5728–5739, 2022.
  6. K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 6360–6376, 2021.
  7. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE transactions on image processing (TIP), pp. 3142–3155, 2017.
  8. K. Zhang, W. Zuo, and L. Zhang, “Ffdnet: Toward a fast and flexible solution for cnn-based image denoising,” IEEE Transactions on Image Processing (TIP), pp. 4608–4622, 2018.
  9. M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” International Conference on Learning Representations (ICLR), pp. 1–14, 2016.
  10. C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  11. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in Neural Information Processing Systems (NeurIPS), pp. 1–9, 2014.
  12. O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8183–8192, 2018.
  13. A. Maleky, S. Kousha, M. S. Brown, and M. A. Brubaker, “Noise2noiseflow: Realistic camera noise modeling without clean images,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17 632–17 641, 2022.
  14. G. Jang, W. Lee, S. Son, and K. M. Lee, “C2n: Practical generative noise modeling for real-world denoising,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2350–2359, 2021.
  15. J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems (NeurIPS), pp. 6840–6851, 2020.
  16. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” International Conference on Machine Learning (ICML), pp. 2256–2265, 2015.
  17. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” International Conference on Learning Representations (ICLR), pp. 1–12, 2021.
  18. H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Srdiff: Single image super-resolution with diffusion probabilistic models,” Neurocomputing, pp. 47–59, 2022.
  19. C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 1–12, 2022.
  20. A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11 461–11 471, 2022.
  21. Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van Gool, “Denoising diffusion models for plug-and-play image restoration,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219–1229, 2023.
  22. J. Whang, M. Delbracio, H. Talebi, C. Saharia, A. G. Dimakis, and P. Milanfar, “Deblurring via stochastic refinement,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16 293–16 303, 2022.
  23. P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” Advances in Neural Information Processing Systems (NeurIPS), pp. 8780–8794, 2021.
  24. Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6228–6237, 2018.
  25. M. Elad and M. Aharon, “Image denoising via sparse and redundant representations over learned dictionaries,” IEEE Transactions on Image processing (TIP), pp. 3736–3745, 2006.
  26. A. Buades, B. Coll, and J. M. Morel, “A non-local algorithm for image denoising,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 60–65, 2005.
  27. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Transactions on image processing (TIP), pp. 2080–2095, 2007.
  28. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D Nonlinear Phenomena, pp. 259–268, 1992.
  29. Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17 683–17 693, 2022.
  30. H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, “Pre-trained image processing transformer,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12 299–12 310, 2021.
  31. S. Cheng, Y. Wang, H. Huang, D. Liu, H. Fan, and S. Liu, “Nbnet: Noise basis learning for image denoising with subspace projection,” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 4896–4906, 2021.
  32. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 14 821–14 831, 2021.
  33. A. Krull, T.-O. Buchholz, and F. Jug, “Noise2void-learning denoising from single noisy images,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2129–2137, 2019.
  34. J. Batson and L. Royer, “Noise2self: Blind denoising by self-supervision,” International Conference on Machine Learning (ICML), pp. 524–533, 2019.
  35. W. Lee, S. Son, and K. M. Lee, “Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17 725–17 734, 2022.
  36. Z. Wang, Y. Fu, J. Liu, and Y. Zhang, “LG-BPN: Local and global blind-patch network for self-supervised real-world denoising,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18 156–18 165, 2023.
  37. T. Pang, H. Zheng, Y. Quan, and H. Ji, “Recorrupted-to-recorrupted: Unsupervised deep learning for image denoising,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2043–2052, 2021.
  38. S. Osher, M. Burger, D. Goldfarb, J. Xu, and W. Yin, “An iterative regularization method for total variation-based image restoration,” Multiscale Modeling & Simulation, pp. 460–489, 2005.
  39. W. Dong, X. Li, L. Zhang, and G. Shi, “Sparsity-based image denoising via dictionary learning and structural clustering,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 457–464, 2011.
  40. W. Dong, P. Wang, W. Yin, G. Shi, F. Wu, and X. Lu, “Denoising prior driven deep neural network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 2305–2318, 2019.
  41. D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” International Conference on Machine Learning (ICML), pp. 1530–1538, 2015.
  42. I. Marras, G. G. Chrysos, I. Alexiou, G. Slabaugh, and S. Zafeiriou, “Reconstructing the noise manifold for image denoising,” arXiv preprint arXiv:2002.04147, pp. 1–18, 2020.
  43. A. Abdelhamed, M. A. Brubaker, and M. S. Brown, “Noise flow: Noise modeling with conditional normalizing flows,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3165–3173, 2019.
  44. K.-C. Chang, R. Wang, H.-J. Lin, Y.-L. Liu, C.-P. Chen, Y.-L. Chang, and H.-T. Chen, “Learning camera-aware noise models,” Proceedings of the European Conference on Computer Vision (ECCV), pp. 343–358, 2020.
  45. Y. Cai, X. Hu, H. Wang, Y. Zhang, H. Pfister, and D. Wei, “Learning to generate realistic noisy images via pixel-level noise-aware adversarial training,” Advances in Neural Information Processing Systems (NeurIPS), pp. 3259–3270, 2021.
  46. K. Lin, T. H. Li, S. Liu, and G. Li, “Real photographs denoising with noise domain adaptation and attentive generative adversarial network,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 3155–3164, 2019.
  47. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10 684–10 695, 2022.
  48. Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” Advances in Neural Information Processing Systems (NeurIPS), pp. 11 895–11 907, 2019.
  49. V. Kulikov, S. Yadin, M. Kleiner, and T. Michaeli, “Sinddm: A single image denoising diffusion model,” International Conference on Machine Learning (ICLR), pp. 17 920–17 930, 2023.
  50. B. Kawar, M. Elad, S. Ermon, and J. Song, “Denoising diffusion restoration models,” Advances in Neural Information Processing Systems (NeurIPS), pp. 1–13, 2022.
  51. Y. Wang, J. Yu, and J. Zhang, “Zero-shot image restoration using denoising diffusion null-space model,” International Conference on Learning Representations (ICLR), pp. 1–12, 2022.
  52. Y. Xie, M. Yuan, B. Dong, and Q. Li, “Diffusion model for generative image denoising,” arXiv preprint arXiv:2302.02398, 2023.
  53. D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  54. J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, pp. 1–12, 2020.
  55. C. Chen, Q. Chen, M. N. Do, and V. Koltun, “Seeing motion in the dark,” 2019, pp. 3185–3194.
  56. D. Zheng, S. H. Tan, X. Zhang, Z. Shi, K. Ma, and C. Bao, “An unsupervised deep learning approach for real-world image denoising,” International Conference on Learning Representations (ICLR), pp. 1–23, 2021.
  57. S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein et al., “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine learning, pp. 1–122, 2011.
  58. C. M. Stein, “Estimation of the mean of a multivariate normal distribution,” The annals of Statistics, pp. 1135–1151, 1981.
  59. S. Soltanayev and S. Y. Chun, “Training deep learning based denoisers without ground truth data,” Advances in Neural Information Processing Systems (NeurIPS), pp. 3261–3271, 2018.
  60. S. Ramani, T. Blu, and M. Unser, “Monte-carlo sure: A black-box optimization of regularization parameters for general denoising algorithms,” IEEE Transactions on image processing, pp. 1540–1554, 2008.
  61. C. Liu, W. Freeman, R. Szeliski, and S. B. Kang, “Noise estimation from a single image,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 901–908, 2006.
  62. A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian, “Practical poissonian-gaussian noise modeling and fitting for single-image raw-data,” IEEE Transactions on Image Processing (TIP), pp. 1737–1754, 2008.
  63. D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 416–423, 2001.
  64. R. Franzen, “Kodak lossless true color image suite,” source: http://r0k. us/graphics/kodak, pp. 1–1, 1999.
  65. L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” Journal of Electronic imaging, pp. 023 016–023 016, 2011.
  66. J. B. Huang, A. Singh, and N. Ahuja, “Single image super-resolution from transformed self-exemplars,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5197–5206, 2015.
  67. C. Tian, Y. Xu, and W. Zuo, “Image denoising using deep cnn with batch renormalization,” Neural Networks, pp. 461–473, 2020.
  68. Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” arXiv preprint arXiv:1903.10082, pp. 1–13, 2019.
  69. Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual dense network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 2480–2495, 2020.
  70. H. Zhao, Y. Gou, B. Li, D. Peng, J. Lv, and X. Peng, “Comprehensive and delicate: An efficient transformer for image restoration,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14 122–14 132, 2023.
  71. Y. Peng, L. Zhang, S. Liu, X. Wu, Y. Zhang, and X. Wang, “Dilated residual networks with symmetric skip connection for image denoising,” Neurocomputing, pp. 67–76, 2019.
  72. D. Guo, S. Shamai, and S. Verdú, “Mutual information and minimum mean-square error in gaussian channels,” IEEE transactions on information theory, pp. 1261–1282, 2005.
  73. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International Journal of Computer Vision (IJCV), pp. 211–252, 2015.
  74. D. Liu, H. Zhang, and Z. Xiong, “On the classification-distortion-perception tradeoff,” Advances in Neural Information Processing Systems (NeurIPS), pp. 1–8, 2019.
  75. J. Zhang, Y. Zhang, J. Gu, Y. Zhang, L. Kong, and X. Yuan, “Accurate image restoration with attention retractable transformer,” International Conference on Learning Representations (ICLR), pp. 1–13, 2023.
  76. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing (TIP), pp. 600–612, 2004.
  77. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595, 2018.
  78. S. Nam, Y. Hwang, Y. Matsushita, and S. J. Kim, “A holistic approach to cross-channel image noise modeling and its application to image denoising,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1683–1691, 2016.
  79. J. Xu, H. Li, Z. Liang, D. Zhang, and L. Zhang, “Real-world noisy image denoising: A new benchmark,” arXiv preprint arXiv:1804.02603, pp. 1–13, 2018.
  80. Y. Zhang, Y. Zhu, E. Nichols, Q. Wang, S. Zhang, C. Smith, and S. Howard, “A poisson-gaussian denoising dataset with real fluorescence microscopy images,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11 710–11 718, 2019.
  81. Z. Yue, Q. Zhao, L. Zhang, and D. Meng, “Dual adversarial network: Toward real-world noise removal and noise generation,” Proceedings of the European Conference on Computer Vision (ECCV), pp. 41–58, 2020.
  82. S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Learning enriched features for real image restoration and enhancement,” Proceedings of the European Conference on Computer Vision (ECCV), pp. 492–511, 2020.
  83. C. Ren, X. He, C. Wang, and Z. Zhao, “Adaptive consistency prior based deep network for image denoising,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8596–8606, 2021.
  84. L. Chen, X. Chu, X. Zhang, and J. Sun, “Simple baselines for image restoration,” Proceedings of the European conference on computer vision (ECCV), pp. 17–33, 2022.
  85. Y. Quan, M. Chen, T. Pang, and H. Ji, “Self2self with dropout: Learning self-supervised denoising from single image,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1890–1898, 2020.
  86. Y. Mansour and R. Heckel, “Zero-shot noise2noise: Efficient image denoising without any data,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14 018–14 027, 2023.
  87. J. Cheng, T. Liu, and S. Tan, “Score priors guided deep variational inference for unsupervised real-world single image denoising,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1–12, 2023.
  88. A. Abdelhamed, S. Lin, and M. S. Brown, “A high-quality denoising dataset for smartphone cameras,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1692–1700, 2018.
  89. G. Chen, F. Zhu, and P. Ann Heng, “An efficient statistical method for image noise level estimation,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 477–485, 2015.
  90. H. Feng, L. Wang, Y. Wang, H. Fan, and H. Huang, “Learnability enhancement for low-light raw image denoising: A data perspective,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 46, no. 1, pp. 370–387, 2024.
  91. A. Bansal, E. Borgnia, H.-M. Chu, J. Li, H. Kazemi, F. Huang, M. Goldblum, J. Geiping, and T. Goldstein, “Cold diffusion: Inverting arbitrary image transforms without noise,” Advances in Neural Information Processing Systems (NeurIPS), pp. 41 259–41 282, 2023.
  92. A. Jolicoeur-Martineau, K. Fatras, K. Li, and T. Kachman, “Diffusion models with location-scale noise,” arXiv preprint arXiv:2304.05907, pp. 1–5, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Tong Li (197 papers)
  2. Hansen Feng (5 papers)
  3. Lizhi Wang (37 papers)
  4. Zhiwei Xiong (83 papers)
  5. Hua Huang (70 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now