Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning from small data sets: Patch-based regularizers in inverse problems for image reconstruction (2312.16611v1)

Published 27 Dec 2023 in cs.CV, cs.LG, eess.IV, and math.PR

Abstract: The solution of inverse problems is of fundamental interest in medical and astronomical imaging, geophysics as well as engineering and life sciences. Recent advances were made by using methods from machine learning, in particular deep neural networks. Most of these methods require a huge amount of (paired) data and computer capacity to train the networks, which often may not be available. Our paper addresses the issue of learning from small data sets by taking patches of very few images into account. We focus on the combination of model-based and data-driven methods by approximating just the image prior, also known as regularizer in the variational model. We review two methodically different approaches, namely optimizing the maximum log-likelihood of the patch distribution, and penalizing Wasserstein-like discrepancies of whole empirical patch distributions. From the point of view of Bayesian inverse problems, we show how we can achieve uncertainty quantification by approximating the posterior using Langevin Monte Carlo methods. We demonstrate the power of the methods in computed tomography, image super-resolution, and inpainting. Indeed, the approach provides also high-quality results in zero-shot super-resolution, where only a low-resolution image is available. The paper is accompanied by a GitHub repository containing implementations of all methods as well as data examples so that the reader can get their own insight into the performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (112)
  1. Operator discretization library (ODL), 2018.
  2. J. Adler and O. Öktem. Deep Bayesian inversion. arXiv preprint arXiv:1811.05910, 2018.
  3. PatchNR: Learning from very few images by patch normalizing flow regularization. Inverse Problems, 39(6):064006, 2023.
  4. Conditional generative models are provably robust: pointwise guarantees for Bayesian inverse problems. Transactions on Machine Learning Research, 2023.
  5. F. Altekrüger and J. Hertrich. WPPNets and WPPFlows: The power of Wasserstein patch priors for superresolution. SIAM Journal on Imaging Sciences, 16(3):1033–1067, 2023.
  6. Invertible neural networks versus MCMC for posterior reconstruction in grazing incidence x-ray fluorescence. International Conference on Scale Space and Variational Methods in Computer Vision, page 528–539, 2021.
  7. Analyzing inverse problems with invertible neural networks. International Conference on Learning Representations, 2018.
  8. Guided image generation with conditional invertible neural networks. arXiv preprint arXiv:1907.02392, 2019.
  9. Wasserstein generative adversarial networks. International Conference on Machine Learning, pages 214–223, 2017.
  10. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A completed reference database of lung nodules on CT scans. Medical Physics, 38(2):915–931, 2011.
  11. Solving inverse problems using data-driven models. Acta Numerica, 28:1–174, 2019.
  12. Conditional image generation with score-based diffusion models. arXiv preprint arXiv:2111.13606, 2021.
  13. Invertible residual networks. International Conference on Machine Learning, pages 573–582, 2019.
  14. Understanding and mitigating exploding inverses in invertible neural networks. ArXiv 2006.09347, 2020.
  15. M. Benning and M. Burger. Modern regularization methods for inverse problems. Acta Numerica, 27:1–111, 2018.
  16. Introduction to Inverse Problems in Imaging. CRC Press, 2021.
  17. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. British Machine Vision Conference, 2012.
  18. NF-ULA: Langevin Monte Carlo with normalizing flow prior for imaging inverse problems. arXiv preprint arXiv:2304.08342, 2023.
  19. T. Q. Chen and M. Schmidt. Fast patch-based style transfer of arbitrary style. Advances in Neural Information Processing Systems, 2016.
  20. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
  21. Image denoising with generalized Gaussian mixture model patch priors. SIAM Journal on Imaging Sciences, 11(4):2568–2609, 2018.
  22. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39(1):1–22, 1977.
  23. Density estimation using Real NVP. International Conference on Learning Representations, 2016.
  24. Sparsity-based image denoising via dictionary learning and structural clustering. IEEE Conference on Computer Vision and Pattern Recognition, pages 457–464, 2011.
  25. Nonparametric generative modeling with conditional sliced-Wasserstein flows. International Conference on Machine Learning, pages 8565–8584, 2023.
  26. A. Elnekave and Y. Weiss. Generating natural images with direct patch distributions matching. European Conference on Computer Vision, pages 544–560, 2022.
  27. Regularization of inverse problems, volume 375. Springer Science & Business Media, 1996.
  28. Interpolating between optimal transport and MMD using Sinkhorn divergences. International Conference on Artificial Intelligence and Statistics, pages 2681–2690, 2019.
  29. R. Friedman and Y. Weiss. Posterior sampling for image restoration using explicit patch priors. arXiv preprint arXiv:2104.09895, 2021.
  30. Texture synthesis using convolutional neural networks. Advances in Neural Information Processing Systems, 28, 2015.
  31. Image style transfer using convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
  32. Stochastic optimization for large-scale optimal transport. Advances in Neural Information Processing Systems, 29, 2016.
  33. Learning generative models with Sinkhorn divergences. International Conference on Artificial Intelligence and Statistics, pages 1608–1617, 2018.
  34. Learned patch-based regularization for inverse problems in imaging. IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, pages 211–215, 2019.
  35. Super-resolution from a single image. IEEE International Conference on Computer Vision, pages 349–356, 2009.
  36. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
  37. Bayesian Gaussian mixture linear inversion for geophysical inverse problems. Mathematical Geosciences, 49(4):493–515, 2017.
  38. Drop the GAN: In defense of patches nearest neighbors as single image generative models. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13460–13469, 2022.
  39. A. Griewank and A. Walther. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM, 2008.
  40. Flow-GAN: Combining maximum likelihood and adversarial learning in generative models. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018.
  41. Improved training of Wasserstein GANs. Advances in Neural Information Processing Systems, 30, 2017.
  42. Optimal patch assignment for statistically constrained texture synthesis. International Conference on Scale Space and Variational Methods in Computer Vision, pages 172–183, 2017.
  43. Posterior sampling based on gradient flows of the MMD with negative distance kernel. arXiv preprint arXiv:2310.03054, 2023.
  44. Stochastic normalizing flows for inverse problems: A Markov chains viewpoint. SIAM/ASA Journal on Uncertainty Quantification, 10(3):1162–1190, 2022.
  45. Generalized Normalizing Flows via Markov Chains. Elements in Non-local Data Interactions: Foundations and Applications. Cambridge University Press, 2023.
  46. P. Hagemann and S. Neumayer. Stabilizing invertible neural networks using mixture models. Inverse Problems, 37(8), 2021.
  47. Alternatives to the EM algorithm for ML estimation of location, scatter matrix, and degree of freedom of the Student-t distribution. Numerical Algorithms, 87(1):77–118, 2021.
  48. Image denoising using asymmetric Gaussian mixture models. IEEE International Symposium in Sensing and Instrumentation in IoT Era, pages 1–4, 2018.
  49. Wasserstein patch prior for image superresolution. IEEE Transactions on Computational Imaging, 8:693–704, 2022.
  50. PCA reduced Gaussian mixture models with application in superresolution. Inverse Problems and Imaging, 16(2):341–366, 2022.
  51. A. Hore and D. Ziou. Image quality metrics: PSNR vs. SSIM. International Conference on Pattern Recognition, pages 2366–2369, 2010.
  52. High-dimensional mixture models for unsupervised image denoising (HDMI). SIAM Journal on Imaging Sciences, 11(4):2815–2846, 2018.
  53. Wasserstein generative models for patch-based texture synthesis. International Conference on Scale Space and Variational Methods in Computer Vision, pages 269–280, 2021.
  54. A generative model for texture synthesis based on optimal transport between feature distributions. Journal of Mathematical Imaging and Vision, 65(1):4–28, 2023.
  55. Tails of Lipschitz triangular flows. International Conference on Machine Learning, 119:4673–4681, 2020.
  56. D. P. Kingma and J. Ba. Adam: a method for stochastic optimization. International Conference on Learning Representations, 2015.
  57. D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114, 2013.
  58. J. Latz. On the well-posedness of Bayesian inverse problems. SIAM/ASA Journal on Uncertainty Quantification, 8(1):451–482, 2020.
  59. Bayesian imaging using Plug & Play priors: When Langevin meets Tweedie. SIAM Journal on Imaging Sciences, 15(2):701–737, 2022.
  60. A nonlocal denoising algorithm for manifold-valued images using second order statistics. SIAM Journal on Imaging Sciences, 10(1):416–448, 2017.
  61. A nonlocal Bayesian image denoising algorithm. SIAM Journal on Imaging Sciences, 6(3):1665–1688, 2013.
  62. Secrets of image denoising cuisine. Acta Numerica, 21:475–576, 2012.
  63. LoDoPaB-CT, a benchmark dataset for low-dose computed tomography reconstruction. Scientific Data, 8(109), 2021.
  64. Demystifying neural style transfer. International Joint Conference on Artificial Intelligence, page 2230–2236, 2017.
  65. Real-time texture synthesis by patch-based sampling. ACM Transactions on Graphics, 20(3):127–150, 2001.
  66. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatics, 10(1):1–9, 2018.
  67. Wasserstein generative learning of conditional distribution. arXiv preprint arXiv:2112.10039, 2021.
  68. Adversarial regularizers in inverse problems. Advances in Neural Information Processing Systems, 2018.
  69. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. IEEE International Conference on Computer Vision, pages 416–423, 2001.
  70. Semi-unbalanced regularized optimal transport for image restoration. European Signal Processing Conference, pages 466–470, 2023.
  71. R. Neal. Bayesian learning via stochastic dynamics. Advances in Neural Information Processing Systems, 5, 1992.
  72. S. Neumayer and G. Steidl. From optimal transport to discrepancy. Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging, pages 1–36, 2021.
  73. Image super-resolution with PCA reduced generalized Gaussian mixture models in materials science. Inverse Problems and Imaging, 17(6):1165–1192, 2023.
  74. Low dimensional manifold model for image processing. SIAM Journal on Imaging Sciences, 10(4):1669–1690, 2017.
  75. V. Papyan and M. Elad. Multi-scale patch-based image restoration. IEEE Transactions on Image Processing, 25(1):249–261, 2015.
  76. Accelerating GMM-based patch priors for image restoration: Three ingredients for a 100x speed-up. IEEE Transactions on Image Processing, 28(2):687–698, 2018.
  77. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
  78. Non-local regularization of inverse problems. European Conference on Computer Vision, pages 57–68, 2008.
  79. Learning local regularization for variational image restoration. International Conference on Scale Space and Variational Methods in Computer Vision, pages 358–370, 2021.
  80. J. Radon. On the determination of functions from their integral values along certain manifolds. IEEE Transactions on Medical Imaging, 5(4):170–176, 1986.
  81. Quality assessment for super-resolution image enhancement. IEEE International Conference on Image Processing, pages 2017–2020, 2006.
  82. General state space Markov chains and MCMC algorithms. Probability Surveys, 1:20 – 71, 2004.
  83. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341 – 363, 1996.
  84. S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2:860–867, 2005.
  85. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, 1992.
  86. Can push-forward generative models fit multimodal distributions? Advances in Neural Information Processing Systems, 2022.
  87. F. Santambrogio. Optimal Transport for Applied Mathematicians Calculus of Variations, PDEs, and Modeling, volume 55. Springer, 2015.
  88. Image quality assessment through FSIM, SSIM, MSE and PSNR—A comparative study. Journal of Computer and Communications, 7(3):8–18, 2019.
  89. Unbalanced optimal transport, from theory to numerics. Handbook of Numerical Analysis, 24:407–471, 2023.
  90. Singan: Learning a generative model from a single natural image. IEEE/CVF International Conference on Computer Vision, pages 4570–4580, 2019.
  91. InGAN: Capturing and retargeting the ”DNA” of a natural image. IEEE/CVF International Conference on Computer Vision, pages 4492–4501, 2019.
  92. “Zero-shot” super-resolution using deep internal learning. IEEE Conference on Computer Vision and Pattern Recognition, pages 3118–3126, 2018.
  93. Natural image statistics and neural representation. Annual Review of Neuroscience, 24(1):1193–1216, 2001.
  94. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
  95. Learning structured output representation using deep conditional generative models. Advances in Neural Information Processing Systems, 28:3483–3491, 2015.
  96. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415–1428, 2021.
  97. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
  98. B. Sprungk. On the local Lipschitz stability of Bayesian inverse problems. Inverse Problems, 36(5), 2020.
  99. A. M. Stuart. Inverse problems: A Bayesian perspective. Acta Numerica, 19:451–559, 2010.
  100. A. N. Tikhonov. On the solution of ill-posed problems and the method of regularization. Doklady Akademii Nauk, 151(3):501–504, 1963.
  101. A. Torralba and A. Oliva. Statistics of natural image categories. Network: Computation in Neural Systems, 14(3):391, 2003.
  102. On-line tools for microscopic and macroscopic monitoring of microwave processing. Physica B: Condensed Matter, 398(2):191–195, 2007.
  103. SURE guided Gaussian mixture image denoising. SIAM Journal on Imaging Sciences, 6(2):999–1034, 2013.
  104. M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. International Conference on Machine Learning, page 681–688, 2011.
  105. Learning likelihoods with conditional normalizing flows. arXiv preprint arXiv:1912.00042, 2019.
  106. Stochastic normalizing flows. Advances in Neural Information Processing Systems, 33, 2020.
  107. EPLL image denoising with multi-feature dictionaries. Digital Signal Processing, 137, 2023.
  108. Explicit diffusion of Gaussian mixture model based image priors. International Conference on Scale Space and Variational Methods in Computer Vision, pages 3–15, 2023.
  109. FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 20(8):2378–2386, 2011.
  110. The unreasonable effectiveness of deep features as a perceptual metric. IEEE Conference on Computer Vision and Pattern Recognition, pages 586–595, 2018.
  111. M. Zontak and M. Irani. Internal statistics of a single natural image. IEEE Conference on Computer Vision and Pattern Recognition, pages 977–984, 2011.
  112. D. Zoran and Y. Weiss. From learning models of natural image patches to whole image restoration. International Conference on Computer Vision, pages 479–486, 2011.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now