Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generative Plug and Play: Posterior Sampling for Inverse Problems (2306.07233v1)

Published 12 Jun 2023 in cs.CV and eess.IV

Abstract: Over the past decade, Plug-and-Play (PnP) has become a popular method for reconstructing images using a modular framework consisting of a forward and prior model. The great strength of PnP is that an image denoiser can be used as a prior model while the forward model can be implemented using more traditional physics-based approaches. However, a limitation of PnP is that it reconstructs only a single deterministic image. In this paper, we introduce Generative Plug-and-Play (GPnP), a generalization of PnP to sample from the posterior distribution. As with PnP, GPnP has a modular framework using a physics-based forward model and an image denoising prior model. However, in GPnP these models are extended to become proximal generators, which sample from associated distributions. GPnP applies these proximal generators in alternation to produce samples from the posterior. We present experimental simulations using the well-known BM3D denoiser. Our results demonstrate that the GPnP method is robust, easy to implement, and produces intuitively reasonable samples from the posterior for sparse interpolation and tomographic reconstruction. Code to accompany this paper is available at https://github.com/gbuzzard/generative-pnp-allerton .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (21)
  1. S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2013.   IEEE, 2013, pp. 945–948.
  2. S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, “Plug-and-play priors for bright field electron tomography and sparse interpolation,” IEEE Transactions on Computational Imaging, vol. 2, no. 4, pp. 408–423, Dec 2016.
  3. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Transactions on Image Processing, vol. 16, no. 8, pp. 2080–2095, 2007.
  4. C. J. Pellizzari, T. J. Bate, K. P. Donnelly, G. T. Buzzard, C. A. Bouman, and M. F. Spencer, “Coherent plug-and-play artifact removal: Physics-based deep learning for imaging through aberrations,” Optics and Lasers in Engineering, vol. 164, 2023.
  5. S. Majee, T. Balke, C. A. J. Kemp, G. T. Buzzard, and C. A. Bouman, “Multi-slice fusion for sparse-view and limited-angle 4d ct reconstruction,” IEEE Transactions on Computational Imaging, vol. 7, 2021.
  6. G. T. Buzzard and C. A. B. Stanley H. Chan, Suhas Sreehari, “Plug-and-play unplugged: Optimization-free reconstruction using consensus equilibrium,” SIAM Journal on Imaging Sciences, vol. 11, no. 3, pp. 2001–2020, 2018.
  7. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Advances in Neural Information Processing Systems (NeurIPS, vol. 27, 2014.
  8. D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in International Conference on Learning Representations (ICLR), 2014.
  9. M. Mirza and S. Osindero, “Conditional generative adversarial nets,” in International Conference on Learning Representations (ICLR), 2014. [Online]. Available: http://arxiv.org/abs/1411.1784
  10. M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein generative adversarial networks,” in International Conference on Learning Representations (ICLR), 2017.
  11. Y. Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” in Advances in Neural Information Processing Systems (NeurIPS), 2019, pp. 11 895–11 907.
  12. P. Vincent, “A connection between score matching and denoising autoencoders,” Neural Computation, vol. 23, no. 7, p. 1661–1674, 2011.
  13. Y. Song and S. Ermon, “Improved techniques for training score-based generative models,” in Advances in Neural Information Processing Systems (NeurIPS), 2020.
  14. U. Grenander and M. Miller, “Stochastic relaxation, gibbs distributions, and the bayesian restoration of images,” Journal of the Royal Statistical Society B, vol. 56, no. 4, pp. 549–581, 1994.
  15. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Robust compressed sensing MRI with deep generative priors,” in International Conference on Learning Representations (ICLR), 2021.
  16. B. T. Feng, J. Smith, M. Rubinstein, H. Chang, K. L. Bouman, and W. T. Freeman, “Score-based diffusion models as principled priors for inverse imaging,” 2017.
  17. A. Jalal, M. Arvinte, G. Daras, E. Price, A. G. Dimakis, and J. I. Tamir, “Robust compressed sensing MRI with deep generative priors,” in Advances in Neural Information Processing Systems (NeurIPS), 2021.
  18. Y. Song, L. Shen, L. Xing, and S. Ermon, “Solving inverse problems in medical imaging with score-based generative models,” in International Conference on Learning Representations (ICLR), 2022.
  19. H. Chung, J. Kim, M. T. Mccann, M. L. Klasky, and J. C. Ye, “Diffusion posterior sampling for general noisy inverse problems,” in International Conference on Learning Representations (ICLR), 2023.
  20. S. D. Team, “Super-Voxel Model Based Iterative Reconstruction (SVMBIR),” Software library available from https://github.com/cabouman/svmbir, 2020.
  21. S. Geman and D. Geman, “Stochastic relaxation, gibbs distributions, and the bayesian restoration of images,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. PAMI-6, no. 6, pp. 721–741, 1984.
Citations (7)
View on Semantic Scholar

Summary

  • The paper presents a framework (GPnP) that transforms deterministic PnP into a stochastic process by generating multiple plausible reconstructions from a single measurement.
  • It leverages proximal distributions and generators to create reversible Markov chains that asymptotically converge to the intended target posterior.
  • Experimental validations show its effectiveness in tasks like sparse interpolation and tomographic reconstruction using adapted denoising algorithms.

Exploring the Depths of Generative Plug-and-Play for Inverse Problems

Introduction to Generative Plug-and-Play

The recent manuscript entitled "Generative Plug and Play: Posterior Sampling for Inverse Problems" introduces a novel framework, Generative Plug-and-Play (GPnP), aimed at addressing inverse problems through the lens of generative modeling. This work extends the capabilities of the traditional Plug-and-Play (PnP) method, which has been widely recognized for its ability to integrate image denoisers as implicit prior models within an iterative reconstruction process. The key innovation of GPnP lies in its incorporation of probabilistic sampling strategies, enabling the generation of multiple plausible reconstructions from a single set of measurements. This marks a significant paradigm shift from deterministic to stochastic methods in the field of inverse problem solving.

Theoretical Underpinnings of GPnP

The manuscript proposes a sophisticated mathematical framework to underpin the GPnP approach. At the heart of the method are the definitions of proximal distributions and proximal generators, which allow for the sampling from posterior distributions associated with a given inverse problem. The formulation is grounded on the mathematical constructs of energy functions and Markov Chains (MC), leading to a theoretical foundation that ensures the generated samples asymptotically adhere to the target posterior distribution. A pivotal contribution of the paper is the assertion that the sequence generated by applying proximal generators forms a reversible MC with the intended stationary distribution, effectively bridging the gap between deterministic optimization and stochastic sampling in complex inverse problems.

Implementation and Practical Insights

The operationalization of the GPnP framework is thoroughly discussed, with a clear delineation of how to implement both prior and forward model proximal generators. Utilizing the denoising score matching approach, the authors demonstrate how an existing denoiser can be adapted within the GPnP framework to facilitate sampling from the posterior. Moreover, the paper presents a comprehensive algorithmic structure (Algorithm 1 in the manuscript) for executing the GPnP method, from initialization through to the iterative application of proximal generators. This structured exposition not only affirms the feasibility of the GPnP approach but also provides a blueprint for its implementation in practical inverse problem-solving scenarios.

Experimental Validation and Insights

To validate the efficacy and robustness of the GPnP framework, the authors present a series of experimental simulations focusing on sparse interpolation and tomographic reconstruction tasks. The results, showcased through compelling visualizations, underline the method's ability to produce intuitively reasonable samples that effectively capture the variability inherent to the posterior distribution. These experiments underscore the potential of GPnP to provide a richer understanding of possible solutions to inverse problems, beyond the traditional single deterministic reconstruction. Furthermore, the adoption of BM3D as the denoising prior serves to highlight the versatility of the GPnP framework in leveraging existing denoising algorithms for generative sampling purposes.

The Road Ahead: Implications and Future Directions

The introduction of Generative Plug-and-Play heralds a novel direction for the field of inverse problems, particularly in the context of imaging sciences. By enabling the generation of multiple samples from the posterior distribution, GPnP opens up new avenues for quantifying uncertainty, assessing solution variability, and ultimately enhancing decision-making processes based on reconstructed images. Looking forward, it is anticipated that further research will explore the integration of more complex and tailored denoising models within the GPnP framework, potentially improving both the quality of the samples and the efficiency of the sampling process. Additionally, extending the GPnP approach to accommodate higher-dimensional and more intricate inverse problems represents a promising area for future exploration.

Conclusion

The manuscript on Generative Plug-and-Play represents a significant step forward in the integration of generative modeling techniques with traditional inverse problem-solving methodologies. Through rigorous theoretical development and persuasive experimental validation, the work establishes GPnP as a robust and versatile framework for posterior sampling in inverse problems. As the community continues to advance in this direction, the potential for transformative impacts across various domains of science and engineering remains profound.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown