Papers
Topics
Authors
Recent
2000 character limit reached

Wasserstein Distortion: Unifying Fidelity and Realism (2310.03629v2)

Published 5 Oct 2023 in cs.IT, cs.CV, eess.IV, and math.IT

Abstract: We introduce a distortion measure for images, Wasserstein distortion, that simultaneously generalizes pixel-level fidelity on the one hand and realism or perceptual quality on the other. We show how Wasserstein distortion reduces to a pure fidelity constraint or a pure realism constraint under different parameter choices and discuss its metric properties. Pairs of images that are close under Wasserstein distortion illustrate its utility. In particular, we generate random textures that have high fidelity to a reference texture in one location of the image and smoothly transition to an independent realization of the texture as one moves away from this point. Wasserstein distortion attempts to generalize and unify prior work on texture generation, image realism and distortion, and models of the early human visual system, in the form of an optimizable metric in the mathematical sense.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.
  2. I. Avcıbaş, B. Sankur, and K. Sayood, “Statistical evaluation of image quality measures,” Journal of Electronic Imaging, vol. 11, no. 2, pp. 206–223, 2002.
  3. R. Dosselmann and X. D. Yang, “Existing and emerging image quality metrics,” in Canadian Conference on Electrical and Computer Engineering, 2005.   IEEE, 2005, pp. 1906–1913.
  4. A. Hore and D. Ziou, “Image quality metrics: PSNR vs. SSIM,” in 2010 20th International Conference on Pattern Recognition.   IEEE, 2010, pp. 2366–2369.
  5. Z. Wang and A. C. Bovik, “Mean squared error: Love it or leave it? a new look at signal fidelity measures,” IEEE Signal Processing Magazine, vol. 26, no. 1, pp. 98–117, 2009.
  6. A. Buades, B. Coll, and J.-M. Morel, “A review of image denoising algorithms, with a new one,” Multiscale Modeling & Simulation, vol. 4, no. 2, pp. 490–530, 2005.
  7. S. Nah, S. Son, S. Lee, R. Timofte, and K. M. Lee, “NTIRE 2021 challenge on image deblurring,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 149–165.
  8. Y. Kwon, K. I. Kim, J. Tompkin, J. H. Kim, and C. Theobalt, “Efficient learning of image super-resolution and compression artifact removal with semi-local Gaussian processes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1792–1805, 2015.
  9. Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6228–6237.
  10. E. J. Delp and O. R. Mitchell, “Moment preserving quantization (signal processing),” IEEE Transactions on Communications, vol. 39, no. 11, pp. 1549–1558, 1991.
  11. M. Li, J. Klejsa, and W. B. Kleijn, “On distribution preserving quantization,” 2011.
  12. N. Saldi, T. Linder, and S. Yüksel, “Randomized quantization and source coding with constrained output distribution,” IEEE Transactions on Information Theory, vol. 61, no. 1, pp. 91–106, 2014.
  13. O. Rippel and L. Bourdev, “Real-time adaptive image compression,” in International Conference on Machine Learning.   PMLR, 06–11 Aug 2017, pp. 2922–2930. [Online]. Available: https://proceedings.mlr.press/v70/rippel17a.html
  14. M. Tschannen, E. Agustsson, and M. Lucic, “Deep generative models for distribution-preserving lossy compression,” Advances in Neural Information Processing Systems, vol. 31, 2018. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2018/file/801fd8c2a4e79c1d24a40dc735c051ae-Paper.pdf
  15. E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool, “Generative adversarial networks for extreme learned image compression,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 221–231.
  16. F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson, “High-fidelity generative image compression,” Advances in Neural Information Processing Systems, vol. 33, pp. 11 913–11 924, 2020. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/8a50bae297807da9e97722a0b3fd8f27-Paper.pdf
  17. J. Klejsa, G. Zhang, M. Li, and W. B. Kleijn, “Multiple description distribution preserving quantization,” IEEE Transactions on Signal Processing, vol. 61, no. 24, pp. 6410–6422, 2013.
  18. Y. Blau and T. Michaeli, “Rethinking lossy compression: The rate-distortion-perception tradeoff,” in Proceedings of the 36th International Conference on Machine Learning.   PMLR, 09–15 Jun 2019, pp. 675–685. [Online]. Available: https://proceedings.mlr.press/v97/blau19a.html
  19. R. Matsumoto, “Introducing the perception-distortion tradeoff into the rate-distortion theory of general information sources,” IEICE Communications Express, vol. 7, no. 11, pp. 427–431, 2018.
  20. ——, “Rate-distortion-perception tradeoff of variable-length source coding for general information sources,” IEICE Communications Express, vol. 8, no. 2, pp. 38–42, 2019.
  21. L. Theis and A. B. Wagner, “A coding theorem for the rate-distortion-perception function,” in Neural Compression: From Information Theory to Applications – Workshop @ ICLR 2021, 2021. [Online]. Available: https://openreview.net/forum?id=BzUaLGtKecs
  22. K. Chen, H. Zhou, H. Zhao, D. Chen, W. Zhang, and N. Yu, “Distribution-preserving steganography based on text-to-speech generative models,” IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 5, pp. 3343–3356, 2021.
  23. J. Chen, L. Yu, J. Wang, W. Shi, Y. Ge, and W. Tong, “On the rate-distortion-perception function,” IEEE Journal on Selected Areas in Information Theory, 2022.
  24. A. B. Wagner, “The rate-distortion-perception tradeoff: The role of common randomness,” arXiv preprint arXiv:2202.04147, 2022.
  25. Y. Hamdi and D. Gündüz, “The rate-distortion-perception trade-off with side information,” in 2023 IEEE International Symposium on Information Theory (ISIT), 2023, pp. 1056–1061.
  26. G. Zhang, J. Qian, J. Chen, and A. Khisti, “Universal rate-distortion-perception representations for lossy compression,” Advances in Neural Information Processing Systems, vol. 34, pp. 11 517–11 529, 2021. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/5fde40544cff0001484ecae2466ce96e-Paper.pdf
  27. X. Niu, D. Gündüz, B. Bai, and W. Han, “Conditional rate-distortion-perception trade-off,” arXiv preprint arXiv:2305.09318, 2023.
  28. S. Salehkalaibar, B. Phan, A. Khisti, and W. Yu, “Rate-distortion-perception tradeoff based on the conditional perception measure,” in 2023 Biennial Symposium on Communications (BSC).   IEEE, 2023, pp. 31–37.
  29. L. Theis, T. Salimans, M. D. Hoffman, and F. Mentzer, “Lossy compression with Gaussian diffusion,” arXiv preprint arXiv:2206.08889, 2022.
  30. X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “ESRGAN: Enhanced super-resolution generative adversarial networks,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0–0.
  31. S. Gao, Y. Shi, T. Guo, Z. Qiu, Y. Ge, Z. Cui, Y. Feng, J. Wang, and B. Bai, “Perceptual learned image compression with continuous rate adaptation,” in 4th Challenge on Learned Image Compression, Jun 2021.
  32. J. Portilla and E. P. Simoncelli, “A parametric texture model based on joint statistics of complex wavelet coefficients,” International Journal of Computer Vision, vol. 40, pp. 49–70, 2000.
  33. B. Balas, L. Nakano, and R. Rosenholtz, “A summary-statistic representation in peripheral vision explains visual crowding,” Journal of Vision, vol. 9, no. 12, pp. 13–13, 2009.
  34. R. Rosenholtz, “What your visual system sees where you are not looking,” in Human Vision and Electronic Imaging XVI, vol. 7865.   SPIE, 2011, pp. 343–356.
  35. R. Rosenholtz, J. Huang, A. Raj, B. J. Balas, and L. Ilie, “A summary statistic representation in peripheral vision explains visual search,” Journal of Vision, vol. 12, no. 4, pp. 14–14, 2012.
  36. J. Freeman and E. P. Simoncelli, “Metamers of the ventral stream,” Nature Neuroscience, vol. 14, no. 9, pp. 1195–1201, 2011.
  37. Y. Qiu, A. B. Wagner, J. Ballé, and L. Theis, “Wasserstein distortion: Unifying fidelity and realism,” in 2024 58th Annual Conference on Information Sciences and Systems (CISS), 2024.
  38. Y. Qiu, A. B. Wagner, J. Ballé, and L. Theis, “Wasserstein distortion: Unifying fidelity and realism,” 2024. [Online]. Available: https://openreview.net/forum?id=ICDJDL5lmQ
  39. E. P. Simoncelli and W. T. Freeman, “The steerable pyramid: A flexible architecture for multi-scale derivative computation,” in Proceedings., International Conference on Image Processing, vol. 3, 1995, pp. 444–447 vol.3.
  40. I. Ustyuzhaninov, W. Brendel, L. Gatys, and M. Bethge, “What does it take to generate natural textures?” in International Conference on Learning Representations, 2017. [Online]. Available: https://openreview.net/forum?id=BJhZeLsxx
  41. L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” Advances in Neural Information Processing Systems, vol. 28, 2015. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2015/file/a5e00132373a7031000fd987a3c9f87b-Paper.pdf
  42. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf
  43. M. Lucic, K. Kurach, M. Michalski, S. Gelly, and O. Bousquet, “Are GANs created equal? a large-scale study,” Advances in Neural Information Processing Systems, vol. 31, 2018. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2018/file/e46de7e1bcaaced9a54f1e9d0d2f800d-Paper.pdf
  44. B. Liu, Y. Zhu, K. Song, and A. Elgammal, “Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=1Fqg133qRaI
  45. J. Fan, S. Liu, S. Ma, Y. Chen, and H.-M. Zhou, “Scalable computation of monge maps with general costs,” in ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022. [Online]. Available: https://openreview.net/forum?id=rEnGR3VdDW5
  46. I. Olkin and F. Pukelsheim, “The distance between two random vectors with given dispersion matrices,” Linear Algebra and its Applications, vol. 48, pp. 257–263, 1982.
  47. J. Vacher, A. Davila, A. Kohn, and R. Coen-Cagli, “Texture interpolation for probing visual perception,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 22 146–22 157. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/fba9d88164f3e2d9109ee770223212a0-Paper.pdf
  48. F. Pitié, A. Kokaram, and R. Dahyot, “n𝑛nitalic_n-dimensional probability density function transfer and its application to color transfer,” in Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, vol. 2, 2005, pp. 1434–1439.
  49. N. Bonneel, J. Rabin, G. Peyré, and H. Pfister, “Sliced and Radon Wasserstein barycenters of measures,” Journal of Mathematical Imaging and Vision, vol. 51, pp. 22–45, 2015.
  50. G. Tartavel, G. Peyré, and Y. Gousseau, “Wasserstein loss for image synthesis and restoration,” SIAM Journal on Imaging Sciences, vol. 9, no. 4, pp. 1726–1755, 2016.
  51. E. Heitz, K. Vanhoey, T. Chambon, and L. Belcour, “A sliced Wasserstein loss for neural texture synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 9412–9420.
  52. M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” Advances in Neural Information Processing Systems, vol. 26, 2013. [Online]. Available: https://proceedings.neurips.cc/paper/2013/hash/af21d0c97db2e27e13572cbf59eb343d-Abstract.html
  53. A. J. Smola, A. Gretton, and K. Borgwardt, “Maximum mean discrepancy,” in 13th International Conference, ICONIP, 2006, pp. 3–6.
  54. C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Póczos, “MMD GAN: Towards deeper understanding of moment matching network,” Advances in Neural Information Processing Systems, vol. 30, 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/dfd7468ac613286cdbb40872c8ef3b06-Paper.pdf
  55. C.-L. Li, W.-C. Chang, Y. Mroueh, Y. Yang, and B. Poczos, “Implicit kernel learning,” in The 22nd International Conference on Artificial Intelligence and Statistics.   PMLR, 16–18 Apr 2019, pp. 2007–2016. [Online]. Available: https://proceedings.mlr.press/v89/li19f.html
  56. Y. Rubner, C. Tomasi, and L. J. Guibas, “The earth mover’s distance as a metric for image retrieval,” International Journal of Computer Vision, vol. 40, no. 2, pp. 99–121, 2000.
  57. A. Elnekave and Y. Weiss, “Generating natural images with direct patch distributions matching,” in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2022, pp. 544–560.
  58. A. Houdard, A. Leclaire, N. Papadakis, and J. Rabin, “A generative model for texture synthesis based on optimal transport between feature distributions,” Journal of Mathematical Imaging and Vision, vol. 65, pp. 4–28, 2023.
  59. K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Comparison of full-reference image quality models for optimization of image processing systems,” International Journal of Computer Vision, vol. 129, pp. 1258–1281, 2021.
  60. C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal, “Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization,” ACM Transactions on Mathematical Software (TOMS), vol. 23, no. 4, pp. 550–560, 1997.
  61. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd International Conference on Learning Representations (ICLR 2015).   Computational and Biological Learning Society, 2015. [Online]. Available: https://doi.org/10.48550/arXiv.1409.1556
  62. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition.   IEEE, 2009, pp. 248–255.
  63. M. Jiang, S. Huang, J. Duan, and Q. Zhao, “SALICON: Saliency in context,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1072–1080.
  64. S. O. Dumoulin and B. A. Wandell, “Population receptive field estimates in human visual cortex,” Neuroimage, vol. 39, no. 2, pp. 647–660, 2008.
  65. G. Liu, Y. Gousseau, and G.-S. Xia, “Texture synthesis through convolutional neural networks and spectrum constraints,” in 2016 23rd International Conference on Pattern Recognition (ICPR), 2016, pp. 3234–3239.
  66. X. Snelgrove, “High-resolution multi-scale neural texture synthesis,” in SIGGRAPH Asia 2017 Technical Briefs, 2017.
  67. O. Sendik and D. Cohen-Or, “Deep correlations for texture synthesis,” ACM Transactions on Graphics (ToG), vol. 36, no. 5, pp. 1–15, 2017.
  68. Y. Zhou, Z. Zhu, X. Bai, D. Lischinski, D. Cohen-Or, and H. Huang, “Non-stationary texture synthesis by adversarial expansion,” ACM Transactions on Graphics (ToG), vol. 37, no. 4, Jul 2018.
  69. N. Gonthier, Y. Gousseau, and S. Ladjal, “High-resolution neural texture synthesis with long-range constraints,” Journal of Mathematical Imaging and Vision, vol. 64, no. 5, pp. 478–492, 2022.
  70. R. Chellappa and R. L. Kashyap, “Texture synthesis using 2-D noncausal autoregressive models,” IEEE Trans. on Acoustics, Speech, and Signal Processing, vol. 33, no. 1, 1985.
  71. D. J. Heeger and J. R. Bergen, “Pyramid-based texture analysis/synthesis,” in SIGGRAPH ’95 Proc. of the 22nd Annual Conf. on Computer Graphics and Interactive Techniques, 1995.
  72. A. A. Efros and T. K. Leung, “Texture synthesis by non-parametric sampling,” in Proc. of IEEE Int. Conf. on Computer Vision ICCV, Sep. 1999, pp. 1033–1038.
  73. V. Kwatra, I. Essa, A. Bobick, and N. Kwatra, “Texture optimization for example-based synthesis,” in Proc. of Int. Conf. on Computer Graphics and Interactive Techniques SIGGRAPH, 2005, pp. 795–802.
  74. S. C. Zhu, Y. N. Wu, and D. Mumford, “Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling,” International Journal of Computer Vision, vol. 27, no. 2, pp. 107–126, Nov. 1998.
  75. Y. Qiu and A. B. Wagner, “Low-rate, low-distortion compression with Wasserstein distortion,” arXiv preprint arXiv:2401.16858, 2024.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube