Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Information Decomposition Diagrams Applied beyond Shannon Entropy: A Generalization of Hu's Theorem (2202.09393v4)

Published 18 Feb 2022 in cs.IT and math.IT

Abstract: In information theory, one major goal is to find useful functions that summarize the amount of information contained in the interaction of several random variables. Specifically, one can ask how the classical Shannon entropy, mutual information, and higher interaction information relate to each other. This is answered by Hu's theorem, which is widely known in the form of information diagrams: it relates shapes in a Venn diagram to information functions, thus establishing a bridge from set theory to information theory. In this work, we view random variables together with the joint operation as a monoid that acts by conditioning on information functions, and entropy as a function satisfying the chain rule of information. This abstract viewpoint allows to prove a generalization of Hu's theorem. It applies to Shannon and Tsallis entropy, (Tsallis) Kullback-Leibler Divergence, cross-entropy, Kolmogorov complexity, submodular information functions, and the generalization error in machine learning. Our result implies for Chaitin's Kolmogorov complexity that the interaction complexities of all degrees are in expectation close to Shannon interaction information. For well-behaved probability distributions on increasing sequence lengths, this shows that the per-bit expected interaction complexity and information asymptotically coincide, thus showing a strong bridge between algorithmic and classical information theory.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Infinite Shannon entropy. Journal of Statistical Mechanics: Theory and Experiment, 2013(4), 2013. ISSN 17425468. doi: 10.1088/1742-5468/2013/04/P04010.
  2. Pierre Baudot. The Poincare-Shannon Machine: Statistical Physics and Machine Learning Aspects of Information Cohomology. Entropy, 21(9), sep 2019. ISSN 10994300. doi: 10.3390/E21090881.
  3. Pierre Baudot. On Information Links. arXiv e-prints, page arXiv:2103.02002, 2021. URL http://arxiv.org/abs/2103.02002.
  4. The Homological Nature of Entropy. Entropy, 17(5):3253–3318, 2015. ISSN 10994300. doi: 10.3390/e17053253.
  5. Topological Information Data Analysis. Entropy, 21(9):1–38, 2019. ISSN 10994300. doi: 10.3390/e21090869.
  6. Robert A. Beeler. How to Count: An Introduction to Combinatorics and Its Applications. Springer International Publishing, 2015. ISBN 9783319138435.
  7. Anthony J Bell. THE CO-INFORMATION LATTICE. in Proc. 4th Int. Symp. Independent Component Analysis and Blind Source Separation, pages 921–926, 2003.
  8. Extra-Fine Sheaves and Interaction Decompositions. arXiv e-prints, page arXiv:2009.12646, sep 2020.
  9. Quantifying unique information. Entropy, 16(4):2161–2183, apr 2014. doi: 10.3390/e16042161. URL https://doi.org/10.3390%2Fe16042161.
  10. N. J. Cerf and C. Adami. Entropic Bell Inequalities. Physical Review A - Atomic, Molecular, and Optical Physics, 55(5):3371–3374, 1997. ISSN 10941622. doi: 10.1103/PhysRevA.55.3371.
  11. N. J. Cerf and C. Adami. Quantum extension of conditional probability. Physical Review A - Atomic, Molecular, and Optical Physics, 60(2):893–897, 1999. ISSN 10941622. doi: 10.1103/PhysRevA.60.893.
  12. Gregory. J. Chaitin. Algorithmic Information Theory. Cambridge University Press, oct 1987. ISBN 9780521343060. doi: 10.1017/CBO9780511608858. URL https://www.cambridge.org/core/product/identifier/9780511608858/type/book.
  13. Elements of Information Theory. Wiley-Interscience, 2006. ISBN 9780471241959. doi: 10.1002/047174882X.
  14. A P Dawid. Separoids: A Mathematical Framework for Conditional Independence and Irrelevance. Annals of Mathematics and Artificial Intelligence, 32(1):335–372, 2001. ISSN 1573-7470. doi: 10.1023/A:1016734104787. URL https://doi.org/10.1023/A:1016734104787.
  15. A. Philip Dawid. Conditional Independence in Statistical Theory. Journal of the Royal Statistical Society. Series B (Methodological), 41(1):1–31, 1979. ISSN 00359246. URL http://www.jstor.org/stable/2984718.
  16. A. Philip Dawid. Conditional Independence for Statistical Operations. The Annals of Statistics, 8, 1980. doi: 10.1214/aos/1176345011.
  17. Hubert Dubé. On the Structure of Information Cohomology. Phd dissertation, University of Toronto, 2023.
  18. Jack Edmonds. Submodular functions, matroids, and certain polyhedra, page 11–26. Springer-Verlag, Berlin, Heidelberg, 2003. ISBN 3540005803.
  19. David Ellerman. The Logic of Partitions: Introduction to the Dual of the Logic of Subsets. Review of Symbolic Logic, 3(2):287–350, 2010. ISSN 17550203. doi: 10.1017/S1755020310000018.
  20. Chain Rule for the Quantum Relative Entropy. Physical Review Letters, 124(10):100501, 2020. ISSN 10797114. doi: 10.1103/PhysRevLett.124.100501. URL https://doi.org/10.1103/PhysRevLett.124.100501.
  21. Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy, 20(4):297, apr 2018. doi: 10.3390/e20040297. URL https://doi.org/10.3390%2Fe20040297.
  22. James Fullwood. On a 2-Relative Entropy. Entropy, 24(1):74, dec 2021. doi: 10.3390/e24010074. URL http://arxiv.org/abs/2112.03582.
  23. Peter Gacs. On the Symmetry of Algorithmic Information. Soviet Math. Dokl., 15(January 1974):1477–1480, 1974.
  24. High-Order Interdependencies in the Aging Brain. Brain Connectivity, 11(9):734–744, 2021. ISSN 21580022. doi: 10.1089/brain.2020.0982.
  25. George. Grätzer. Lattice Theory: Foundation. Number July. Springer Basel AG, Basel, 2011. ISBN 9783034800181 3034800185 3034800177 9783034800174. doi: 10.1007/978-3-0348-0018-1. URL http://site.ebrary.com/id/10448298.
  26. Peter D. Grünwald and Paul M B Vitányi. Algorithmic Information Theory. arXiv e-prints, abs/0809.2:arXiv:0809.2754, sep 2008. ISSN 21971765. doi: 10.1007/978-981-15-0739-7˙2.
  27. Te Sun Han. Nonnegative Entropy Measures of Multivariate Symmetric Correlations. Information and Control, 36(2):133–156, 1978. ISSN 0019-9958. doi: https://doi.org/10.1016/S0019-9958(78)90275-9. URL https://www.sciencedirect.com/science/article/pii/S0019995878902759.
  28. Gerhard Hochschild. On the Cohomology Groups of an Associative Algebra. Annals of Mathematics, 46(1):58–67, 1945. ISSN 0003486X. URL http://www.jstor.org/stable/1969145.
  29. Partial quantum information. Nature, 436(7051):673–676, 2005. ISSN 1476-4687. doi: 10.1038/nature03909. URL https://doi.org/10.1038/nature03909http://arxiv.org/abs/quant-ph/0505062{%}0Ahttp://dx.doi.org/10.1038/nature03909.
  30. Kuo Ting Hu. On the Amount of Information. Theory of Probability & Its Applications, 7(4):439–447, 1962. doi: 10.1137/1107041. URL https://doi.org/10.1137/1107041.
  31. Multivariate dependence beyond Shannon information. Entropy, 19(10), 2017. ISSN 10994300. doi: 10.3390/e19100531.
  32. An Introduction to Kolmogorov Complexity and Its Applications. Springer, 1997. ISBN 9783030112974. doi: 10.1007/978-1-4757-3860-5.
  33. Information Decomposition of Target Effects from Multi-Source Interactions: Perspectives on Previous, Current and Future Work. Entropy, 20(4), 2018. ISSN 10994300. doi: 10.3390/e20040307. URL https://www.mdpi.com/1099-4300/20/4/307.
  34. Thomas Murray MacRobert and Thomas John I’Anson Bromwich. An Introduction to the Theory of Infinite Series. Macmillan, London, 1926.
  35. Foundations of Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2 edition, 2018. ISBN 978-0-262-03940-6.
  36. An analysis of approximations for maximizing submodular set functions–I. Math. Program., 14(1):265–294, dec 1978. ISSN 0025-5610. doi: 10.1007/BF01588971. URL https://doi.org/10.1007/BF01588971.
  37. Arthur J. Parzygnat. Towards a functorial description of quantum relative entropy. arXiv e-prints, page arXiv:2105.04059, 2021. doi: 10.1007/978-3-030-80209-7˙60. URL http://arxiv.org/abs/2105.04059{%}0Ahttp://dx.doi.org/10.1007/978-3-030-80209-7{_}60.
  38. Quantifying Synergistic Information Using Intermediate Stochastic Variables †. Entropy, 19(2):85, February 2017. doi: 10.3390/e19020085.
  39. Quantifying High-Order Interdependencies via Multivariate Extensions of the Mutual Information. Physical Review E, 100(3):1–17, 2019. ISSN 24700053. doi: 10.1103/PhysRevE.100.032305.
  40. René L. Schilling. Measures, Integrals and Martingales. Measures, Integrals and Martingales. Cambridge University Press, 2017. ISBN 9781316620243. URL https://books.google.nl/books?id=sdAoDwAAQBAJ.
  41. A Schrijver. Combinatorial Optimization - Polyhedra and Efficiency. Springer, 2003.
  42. Understanding Machine Learning — From Theory to Algorithms. Cambridge University Press, 2014. ISBN 978-1-10-705713-5.
  43. C. E. Shannon. A Mathematical Theory of Communication. The Bell System Technical Journal, 27(3):379–423, 1948. ISSN 15387305. doi: 10.1002/j.1538-7305.1948.tb01338.x.
  44. The Mathematical Theory of Communication. The University of Illinois Press, 1964. ISBN 9780252725487. URL https://pure.mpg.de/rest/items/item{_}2383164{_}3/component/file{_}2383163/content.
  45. Melvin Dale Springer. The Algebra of Random Variables, volume 23 of Wiley series in probability and mathematical statistics. John Wiley & Sons, 1979. doi: 10.2307/1268039.
  46. Causal Markov condition for submodular information measures. COLT 2010 - The 23rd Conference on Learning Theory, pages 464–476, 2010.
  47. Terrence Tao. An Introduction to Measure Theory. American Mathematical Society, 2013. ISBN 9781470409227. URL https://books.google.nl/books?id=SPGJjwEACAAJ.
  48. Constantino Tsallis. Possible Generalization of Boltzmann-Gibbs statistics. Journal of Statistical Physics, 52(1):479–487, 1988. ISSN 1572-9613. doi: 10.1007/BF01016429. URL https://doi.org/10.1007/BF01016429.
  49. V. Vedral. The Role of Relative Entropy in Quantum Information Theory. Reviews of Modern Physics, 74(1):197–234, 2002. ISSN 00346861. doi: 10.1103/RevModPhys.74.197.
  50. Juan Pablo Vigneaux. Topology of Statistical Systems — A Cohomological Approach to Information Theory. Phd dissertation, Université de Paris, 2019.
  51. Juan Pablo Vigneaux. Information Structures and Their Cohomology. Theory and Applications of Categories, 35(38):1476–1529, 2020. ISSN 1201561X.
  52. Juan Pablo Vigneaux. Entropy under Disintegrations. arXiv e-prints, 1:arXiv:2102.09584, 2021. URL http://arxiv.org/abs/2102.09584.
  53. Satosi Watanabe. Information Theoretical Analysis of Multivariate Correlation. IBM Journal of Research and Development, 4(1):66–82, 1960. doi: 10.1147/rd.41.0066.
  54. Nonnegative Decomposition of Multivariate Information. arXiv e-prints, page arXiv:1004.2515, 2010. URL http://arxiv.org/abs/1004.2515.
  55. Decomposing Multivariate Information. Privately Communicated, 2011. URL papers://c941067e-36da-4589-acf1-2f5738fdb5a1/Paper/p989.
  56. Raumond W. Yeung. A New Outlook on Shannon’s Information Measures. IEEE Transactions on Information Theory, 37(3):466–474, 1991. ISSN 21915776. doi: 10.1109/18.79902.
  57. Raymond W Yeung. A First Course in Information Theory. Springer-Verlag, Berlin, Heidelberg, 2002. ISBN 9781461346456.
  58. The Complexity of Finite Objects and the Development of the Concepts of Information and Randomness By Means of the Theory of Algorithms. Russian Mathematical Surveys, 25(6):83–124, 1970. ISSN 0036-0279. doi: 10.1070/rm1970v025n06abeh001269.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now