Papers
Topics
Authors
Recent
Search
2000 character limit reached

Minimum Entropy Coupling with Bottleneck

Published 29 Oct 2024 in cs.LG, cs.IT, and math.IT | (2410.21666v1)

Abstract: This paper investigates a novel lossy compression framework operating under logarithmic loss, designed to handle situations where the reconstruction distribution diverges from the source distribution. This framework is especially relevant for applications that require joint compression and retrieval, and in scenarios involving distributional shifts due to processing. We show that the proposed formulation extends the classical minimum entropy coupling framework by integrating a bottleneck, allowing for a controlled degree of stochasticity in the coupling. We explore the decomposition of the Minimum Entropy Coupling with Bottleneck (MEC-B) into two distinct optimization problems: Entropy-Bounded Information Maximization (EBIM) for the encoder, and Minimum Entropy Coupling (MEC) for the decoder. Through extensive analysis, we provide a greedy algorithm for EBIM with guaranteed performance, and characterize the optimal solution near functional mappings, yielding significant theoretical insights into the structural complexity of this problem. Furthermore, we illustrate the practical application of MEC-B through experiments in Markov Coding Games (MCGs) under rate limits. These games simulate a communication scenario within a Markov Decision Process, where an agent must transmit a compressed message from a sender to a receiver through its actions. Our experiments highlight the trade-offs between MDP rewards and receiver accuracy across various compression rates, showcasing the efficacy of our method compared to conventional compression baseline.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Multiterminal source coding with an entropy-based distortion measure. In 2011 IEEE International Symposium on Information Theory Proceedings, pages 2040–2044. IEEE, 2011.
  2. Multiterminal source coding under logarithmic loss. IEEE Transactions on Information Theory, 60(1):740–761, 2013.
  3. A single-shot approach to lossy source coding under logarithmic loss. IEEE Transactions on Information Theory, 64(1):129–147, 2017.
  4. Mathukumalli Vidyasagar. A metric between probability distributions on finite sets of different cardinalities and applications to order reduction. IEEE Transactions on Automatic Control, 57(10):2464–2477, 2012.
  5. Memoryless representation of markov processes. In 2013 IEEE International Symposium on Information Theory, pages 2294–298. IEEE, 2013.
  6. On the entropy of couplings. Information and Computation, 242:369–382, 2015.
  7. How to find a joint probability distribution of minimum entropy (almost) given the marginals. In 2017 IEEE International Symposium on Information Theory (ISIT), pages 2173–2177. IEEE, 2017.
  8. Maurice Fréchet. Sur les tableaux de corrélation dont les marges sont données. Ann. Univ. Lyon, 3^ e serie, Sciences, Sect. A, 14:53–77, 1951.
  9. Frank Den Hollander. Probability theory: The coupling method. Lecture notes available online (http://websites. math. leidenuniv. nl/probability/lecturenotes/CouplingLectures. pdf), 2012.
  10. Recent developments on the construction of bivariate distributions with fixed marginals. Journal of Statistical Distributions and Applications, 1:1–23, 2014.
  11. Distributions with given marginals and moment problems. Springer Science & Business Media, 2012.
  12. Asymptotic coupling and its applications in information theory. IEEE Transactions on Information Theory, 65(3):1321–1344, 2018.
  13. Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
  14. Entropic causal inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  15. Spencer Compton. A tighter approximation guarantee for greedy minimum entropy coupling. In 2022 IEEE International Symposium on Information Theory (ISIT), pages 168–173. IEEE, 2022.
  16. Minimum-entropy coupling approximation guarantees beyond the majorization barrier. In International Conference on Artificial Intelligence and Statistics, pages 10445–10469. PMLR, 2023.
  17. Inequalities: theory of majorization and its applications. 1979.
  18. Minimum-entropy couplings and their applications. IEEE Transactions on Information Theory, 65(6):3436–3451, 2019.
  19. Cheuk Ting Li. Efficient approximate minimum entropy coupling of multiple probability distributions. IEEE Transactions on Information Theory, 67(8):5259–5268, 2021.
  20. Entropic causal inference: Identifiability and finite sample results. Advances in Neural Information Processing Systems, 33:14772–14782, 2020.
  21. Quantum entropic causal inference. In Quantum Information and Measurement, pages F2C–3. Optica Publishing Group, 2021.
  22. Communicating via markov decision processes. In International Conference on Machine Learning, pages 20314–20328. PMLR, 2022.
  23. Perfectly secure steganography using minimum entropy coupling. arXiv preprint arXiv:2210.14889, 2022.
  24. Rethinking lossy compression: The rate-distortion-perception tradeoff. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 675–685. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/blau19a.html.
  25. Lossy compression with distribution shift as entropy constrained optimal transport. In International Conference on Learning Representations, 2022.
  26. Nicolaas Govert De Bruijn. Asymptotic methods in analysis, volume 4. Courier Corporation, 1981.
  27. Brian Skyrms. Signals: Evolution, learning, and information. OUP Oxford, 2010.
  28. Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
  29. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR, 2023.
  30. Image-to-image translation with conditional adversarial networks. arxiv e-prints. arXiv preprint arXiv:1611.07004, 1611, 2016.
  31. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223–2232, 2017.
  32. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989–1998. Pmlr, 2018.
  33. Toward joint image generation and compression using generative adversarial networks. arXiv preprint arXiv:1901.07838, 2019.
  34. On mutual information maximization for representation learning. arXiv preprint arXiv:1907.13625, 2019.
  35. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670, 2018.
  36. O. L. Mangasarian. Machine Learning via Polyhedral Concave Minimization, pages 175–188. Physica-Verlag HD, Heidelberg, 1996. ISBN 978-3-642-99789-1. doi: 10.1007/978-3-642-99789-1_13. URL https://doi.org/10.1007/978-3-642-99789-1_13.
  37. Michel X Goemans. Spectral graph theory and numerical linear algebra. URL http://www.cs.cmu.edu/afs/cs/user/glmiller/public/Scientific-Computing/F-11/RelatedWork/Goemans-LP-notes.pdf.
  38. Nonlinear optimization by successive linear programming. Management science, 28(10):1106–1120, 1982.
  39. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
  40. Reinforcement learning: An introduction. MIT press, 2018.
  41. Kevin Hanselman. grid-world-rl. https://github.com/kevin-hanselman/grid-world-rl, 2023.
  42. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320):201, 2004.
  43. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
  44. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  45. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 3 tweets with 3 likes about this paper.

Sign Up for Free