Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

The Perception-Robustness Tradeoff in Deterministic Image Restoration (2311.09253v4)

Published 14 Nov 2023 in eess.IV, cs.CV, cs.LG, and eess.SP

Abstract: We study the behavior of deterministic methods for solving inverse problems in imaging. These methods are commonly designed to achieve two goals: (1) attaining high perceptual quality, and (2) generating reconstructions that are consistent with the measurements. We provide a rigorous proof that the better a predictor satisfies these two requirements, the larger its Lipschitz constant must be, regardless of the nature of the degradation involved. In particular, to approach perfect perceptual quality and perfect consistency, the Lipschitz constant of the model must grow to infinity. This implies that such methods are necessarily more susceptible to adversarial attacks. We demonstrate our theory on single image super-resolution algorithms, addressing both noisy and noiseless settings. We also show how this undesired behavior can be leveraged to explore the posterior distribution, thereby allowing the deterministic model to imitate stochastic methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  2. Multi-realism image compression with a conditional generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  22324–22333, June 2023.
  3. Principal uncertainty quantification with spatial correlation for image restoration problems. arXiv, 2024.
  4. Demystifying MMD gans. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=r1lUOzWCW.
  5. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  6. Real-world blind super-resolution via feature matching with implicit high-resolution priors. 2022.
  7. Generative models improve radiomics reproducibility in low dose cts: a simulation study. Physics in Medicine & Biology, 66(16):165002, 2021.
  8. Evaluating robustness of deep image super-resolution against adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  9. Pixel recursive super resolution. 2017. URL https://arxiv.org/abs/1702.00783.
  10. Davison, A. C. Statistical Models. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2003. doi: 10.1017/CBO9780511815850.
  11. Deep generative image models using a laplacian pyramid of adversarial networks. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/paper/2015/file/aa169b49b583a2b5af89203c2b78c67c-Paper.pdf.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=YicbFdNTTy.
  13. Dwikifirdaus, R. Face gender classification with a vision transformer. https://huggingface.co/rizvandwiki/gender-classification-2, 2022. Accessed: 2023-10-23.
  14. The farthest point strategy for progressive image sampling. In Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 2 - Conference B: Computer Vision & Image Processing. (Cat. No.94CH3440-5), pp.  93–97 vol.3, 1994. doi: 10.1109/ICPR.1994.577129.
  15. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1–8, 2021. URL http://jmlr.org/papers/v22/20-451.html.
  16. A theory of the distortion-perception tradeoff in wasserstein space. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  25661–25672. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/d77e68596c15c53c2a33ad143739902d-Paper.pdf.
  17. Frequency separation for real-world super-resolution. In IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2019.
  18. Evaluating adversarial robustness of low dose CT recovery. In Medical Imaging with Deep Learning, 2023. URL https://openreview.net/forum?id=L-N1uAxfQk1.
  19. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf.
  20. The troublesome kernel – on hallucinations, no free lunches and the accuracy-stability trade-off in inverse problems. arXiv, 2023.
  21. Pixcolor: Pixel recursive colorization. Proceedings of the 28th British Machine Vision Conference (BMVC), 2017. URL https://arxiv.org/abs/1705.07208.
  22. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf.
  23. Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph., 35(4), jul 2016. ISSN 0730-0301. doi: 10.1145/2897824.2925974. URL https://doi.org/10.1145/2897824.2925974.
  24. Fast underwater image enhancement for improved visual perception. IEEE Robotics and Automation Letters (RA-L), 5(2):3227–3234, 2020.
  25. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
  26. Real-world super-resolution via kernel estimation and noise injection. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  27. Statistical and Computational Inverse Problems. Springer, Dordrecht, 2005. doi: 10.1007/b138659. URL https://cds.cern.ch/record/1338003.
  28. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  1548–1558, 2021.
  29. Progressive growing of GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hk99zCeAb.
  30. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  4396–4405, 2019. doi: 10.1109/CVPR.2019.00453.
  31. Image denoising with conditional generative adversarial networks (cgan) in low dose chest images. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 954:161914, 2020. ISSN 0168-9002. doi: https://doi.org/10.1016/j.nima.2019.02.041. URL https://www.sciencedirect.com/science/article/pii/S0168900219302293. Symposium on Radiation Measurements and Applications XVII.
  32. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. doi: 10.48550/ARXIV.1412.6980. URL https://arxiv.org/abs/1412.6980.
  33. Kirsch, A. An Introduction to the Mathematical Theory of Inverse Problems, volume 120. 01 2011. ISBN 978-1-4419-8473-9. doi: 10.1007/978-1-4419-8474-6.
  34. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  35. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  36. Adversarial machine learning at scale. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=BJm4T4Kgx.
  37. Improved precision and recall metric for assessing generative models. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc.
  38. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  39. Swinir: Image restoration using swin transformer. arXiv preprint arXiv:2108.10257, 2021.
  40. Aim 2019 challenge on real-world image super-resolution: Methods and results. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp.  3575–3583. IEEE, 2019.
  41. Ntire 2020 challenge on real-world image super-resolution: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  42. Ntire 2021 learning the super-resolution space challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.  596–612, June 2021.
  43. Ntire 2022 challenge on learning the super-resolution space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.  786–797, June 2022.
  44. Unfolding the alternating optimization for blind super resolution. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
  45. Speckle noise reduction in optical coherence tomography images based on edge-sensitive cgan. Biomedical optics express, 9(11):5129–5146, 2018.
  46. High-perceptual quality jpeg decoding via posterior sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.  1272–1282, June 2023.
  47. Deep multi-scale video prediction beyond mean square error. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs/1511.05440.
  48. Which training methods for gans do actually converge? In International Conference on Machine Learning (ICML), 2018.
  49. Conditional generative adversarial nets. CoRR, abs/1411.1784, 2014. URL http://arxiv.org/abs/1411.1784.
  50. Deep generative adversarial residual convolutional networks for real-world super-resolution. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  51. Learned reconstruction methods with convergence guarantees: A survey of concepts and applications. IEEE Signal Processing Magazine, 40(1):164–182, January 2023. ISSN 1053-5888. doi: 10.1109/MSP.2022.3207451. Publisher Copyright: © 1991-2012 IEEE.
  52. Nate Raw. vit-age-classifier (revision 461a4c4). 2023. doi: 10.57967/hf/1259. URL https://huggingface.co/nateraw/vit-age-classifier.
  53. High perceptual quality image denoising with a posterior sampling cgan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1805–1813, 2021.
  54. Reasons for the superiority of stochastic estimators over deterministic ones: Robustness, consistency and perceptual quality. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp.  26474–26494. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/ohayon23a.html.
  55. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  56. Improved techniques for training gans. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_files/paper/2016/file/8a3363abe792db2d8761d6403605aeb7-Paper.pdf.
  57. Hyperextended lightface: A facial attribute analysis framework. In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pp.  1–4. IEEE, 2021. doi: 10.1109/ICEET53442.2021.9659697. URL https://doi.org/10.1109/ICEET53442.2021.9659697.
  58. Spatially-adaptive pixelwise networks for fast image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  14882–14891, June 2021.
  59. Deep learning-based denoising in projection-domain and reconstruction-domain for low-dose myocardial perfusion spect. Journal of Nuclear Cardiology, 30(3):970–985, 2023.
  60. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  2818–2826, 2016. doi: 10.1109/CVPR.2016.308.
  61. Ntire 2017 challenge on single image super-resolution: Methods and results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
  62. Semi-supervised capsule cgan for speckle noise reduction in retinal oct images. IEEE Transactions on Medical Imaging, PP:1–1, 01 2021a. doi: 10.1109/TMI.2020.3048975.
  63. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018a.
  64. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In International Conference on Computer Vision Workshops (ICCVW).
  65. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, September 2018b.
  66. Esrgan: Enhanced super-resolution generative adversarial networks. In The European Conference on Computer Vision Workshops (ECCVW), September 2018c.
  67. Towards real-world blind face restoration with generative facial prior. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021b.
  68. Diversity-sensitive conditional generative adversarial networks. In Proceedings of the International Conference on Learning Representations, 2019.
  69. Sharpness-aware low-dose ct denoising using conditional generative adversarial network. Journal of digital imaging, 31:655–669, 2018.
  70. Deblurring by realistic blurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  71. Designing a practical degradation model for deep blind image super-resolution. In IEEE International Conference on Computer Vision, pp.  4791–4800, 2021.
  72. Colorful image colorization. In ECCV, 2016.
  73. Real-time user-guided image colorization with learned deep priors. ACM Transactions on Graphics (TOG), 9(4), 2017.
  74. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now