Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
121 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration (2303.11435v5)

Published 20 Mar 2023 in eess.IV, cs.CV, and cs.LG

Abstract: Inversion by Direct Iteration (InDI) is a new formulation for supervised image restoration that avoids the so-called "regression to the mean" effect and produces more realistic and detailed images than existing regression-based methods. It does this by gradually improving image quality in small steps, similar to generative denoising diffusion models. Image restoration is an ill-posed problem where multiple high-quality images are plausible reconstructions of a given low-quality input. Therefore, the outcome of a single step regression model is typically an aggregate of all possible explanations, therefore lacking details and realism. The main advantage of InDI is that it does not try to predict the clean target image in a single step but instead gradually improves the image in small steps, resulting in better perceptual quality. While generative denoising diffusion models also work in small steps, our formulation is distinct in that it does not require knowledge of any analytic form of the degradation process. Instead, we directly learn an iterative restoration process from low-quality and high-quality paired examples. InDI can be applied to virtually any image degradation, given paired training data. In conditional denoising diffusion image restoration the denoising network generates the restored image by repeatedly denoising an initial image of pure noise, conditioned on the degraded input. Contrary to conditional denoising formulations, InDI directly proceeds by iteratively restoring the input low-quality image, producing high-quality results on a variety of image restoration tasks, including motion and out-of-focus deblurring, super-resolution, compression artifact removal, and denoising.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (94)
  1. Defocus deblurring using dual-pixel data. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16, pp.  111–126. Springer, 2020.
  2. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  3. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
  4. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=li7qeBbCR1t.
  5. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214–223. PMLR, 2017.
  6. Generalization and equilibrium in generative adversarial nets (GANs). In International Conference on Machine Learning, pp. 224–232. PMLR, 2017.
  7. Blind image deconvolution using deep generative priors. IEEE Transactions on Computational Imaging, 6:1493–1506, 2020.
  8. Cold Diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
  9. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
  10. The Perception-Distortion Tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  6228–6237, 2018.
  11. Compressed sensing using generative models. In International Conference on Machine Learning (ICML), pp. 537–546. PMLR, 2017.
  12. Learning to see in the dark. In IEEE Conference on Computer Vision and Pattern Recognition, pp.  3291–3300, 2018.
  13. Hinet: Half instance normalization network for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.  182–192, June 2021a.
  14. Simple baselines for image restoration. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII, pp.  17–33. Springer, 2022.
  15. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  8628–8638, 2021b.
  16. Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp.  4641–4650, October 2021.
  17. Improving diffusion models for inverse problems using manifold constraints. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
  18. Distribution matching losses can hallucinate features in medical image translation. In International conference on medical image computing and computer-assisted intervention, pp.  529–536. Springer, 2018.
  19. Soft diffusion: Score matching with general corruptions. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
  20. Heavy-tailed denoising score matching. arXiv preprint arXiv:2112.09788, 2021.
  21. Polyblur: Removing mild blur by polynomial reblurring. IEEE Transactions on Computational Imaging, 7:837–848, 2021a.
  22. Projected distribution loss for image enhancement. In 2021 IEEE International Conference on Computational Photography (ICCP), pp.  1–12. IEEE Computer Society, 2021b.
  23. Image super-resolution using deep convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015.
  24. Bradley Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association, 106(496):1602–1614, 2011.
  25. A theory of the Distortion-Perception tradeoff in Wasserstein space. Advances in Neural Information Processing Systems, 34:25661–25672, 2021.
  26. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2414–2423, 2016.
  27. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  28. Iterative α𝛼\alphaitalic_α-(de) blending: a minimalist deterministic diffusion model. arXiv preprint arXiv:2305.03486, 2023.
  29. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  30. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  6840–6851, 2020.
  31. Blurring diffusion models. arXiv preprint arXiv:2209.05557, 2022.
  32. Autoregressive diffusion models. In International Conference on Learning Representations, 2022a.
  33. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867–8887. PMLR, 2022b.
  34. Image-to-image translation with conditional adversarial networks. In IEEE conference on computer vision and pattern recognition, pp.  1125–1134, 2017.
  35. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34:14938–14954, 2021a.
  36. Fairness for image generation with uncertain sensitive attributes. In International Conference on Machine Learning, pp. 4721–4732. PMLR, 2021b.
  37. Beyond in-place corruption: Insertion and deletion in denoising probabilistic models. arXiv preprint arXiv:2107.07675, 2021.
  38. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  39. Plug-and-play methods for integrating physical and learned models in computational imaging. arXiv preprint arXiv:2203.17061, 2022.
  40. SNIPS: Solving noisy inverse problems stochastically. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021a.
  41. Stochastic image denoising by sampling from the posterior distribution. In Proceedings of the International Conference on Computer Vision (ICCV) Workshops, 2021b.
  42. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems, 2022.
  43. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  8183–8192, 2018.
  44. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
  45. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701–737, 2022.
  46. Photo-realistic single image super-resolution using a generative adversarial network. In IEEE conference on computer vision and pattern recognition, pp.  4681–4690, 2017.
  47. Progressive deblurring of diffusion models for coarse-to-fine image synthesis. arXiv preprint arXiv:2207.11192, 2022.
  48. Srdiff: Single image super-resolution with diffusion probabilistic models. arXiv preprint arXiv:2104.14951, 2021.
  49. Details or artifacts: A locally discriminative learning approach to realistic image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5657–5666, 2022.
  50. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp.  136–144, 2017.
  51. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=PqvMRDCJT9t.
  52. I22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTsb: Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.05872, 2023a.
  53. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=XVjTT1nw5z.
  54. Srflow: Learning the super-resolution space with normalizing flow. In European Conference on Computer Vision, pp.  715–732. Springer, 2020.
  55. Ntire 2021 learning the super-resolution space challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  596–612, 2021.
  56. Image restoration with mean-reverting stochastic differential equations. arXiv preprint arXiv:2301.11699, 2023a.
  57. Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  1680–1691, 2023b.
  58. Maintaining natural image statistics with the contextual loss. In Asian Conference on Computer Vision, pp.  427–443. Springer, 2018a.
  59. The contextual loss for image transformation with non-aligned data. In European Conference on Computer Vision (ECCV), pp. 768–783, 2018b.
  60. Denoising diffusion gamma models. arXiv preprint arXiv:2110.05948, 2021.
  61. Deep multi-scale convolutional neural network for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  62. High perceptual quality image denoising with a posterior sampling cgan. In Proceedings of the International Conference on Computer Vision (ICCV) Workshops, 2021.
  63. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39–56, 2020.
  64. Fully unsupervised diversity denoising with convolutional variational autoencoders. In International Conference on Learning Representations, 2020.
  65. Herbert Robbins. An empirical bayes approach to statistics. In Proc. 3rd Berkeley Symp. Math. Statist. Probab., 1956, volume 1, pp.  157–163, 1956.
  66. The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences, 10(4):1804–1844, 2017.
  67. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp.  234–241. Springer, 2015.
  68. Total variation based image restoration with free local constraints. In Proceedings of 1st international conference on image processing, volume 1, pp.  31–35. IEEE, 1994.
  69. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07636, 2021.
  70. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp.  1–10, 2022.
  71. Improved techniques for training GANs. Advances in neural information processing systems, 29, 2016.
  72. Diffusion schr\\\backslash\" odinger bridge matching. arXiv preprint arXiv:2303.16852, 2023.
  73. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256–2265. PMLR, 2015.
  74. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
  75. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
  76. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  77. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
  78. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
  79. Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  80. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  81. Maxim: Multi-axis mlp for image processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5769–5780, 2022.
  82. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp.  945–948. IEEE, 2013.
  83. Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
  84. ESRGAN: enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp.  0–0, 2018.
  85. Blind drifting: Diffusion models with a linear SDE drift term for blind image restoration tasks. In The Symbiosis of Deep Learning and Differential Equations II, 2022. URL https://openreview.net/forum?id=VCLnhfPVEB.
  86. Composing normalizing flows for inverse problems. In International Conference on Machine Learning, pp. 11158–11169. PMLR, 2021.
  87. Deblurring via stochastic refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16293–16303, 2022.
  88. First hitting diffusion models. arXiv preprint arXiv:2209.01170, 2022.
  89. Multi-stage progressive image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  14821–14831, 2021.
  90. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5728–5739, 2022.
  91. Deep stacked hierarchical multi-patch network for image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  92. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  4791–4800, 2021.
  93. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  94. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging, 3(1):47–57, 2016.
Citations (89)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now