Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 148 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 34 tok/s Pro
GPT-5 High 40 tok/s Pro
GPT-4o 101 tok/s Pro
Kimi K2 183 tok/s Pro
GPT OSS 120B 443 tok/s Pro
Claude Sonnet 4.5 35 tok/s Pro
2000 character limit reached

Mutual Information Estimation via Normalizing Flows (2403.02187v3)

Published 4 Mar 2024 in cs.LG, cs.IT, math.IT, and stat.ML

Abstract: We propose a novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data to the target distribution, for which MI is easier to estimate. We additionally explore the target distributions with known closed-form expressions for MI. Theoretical guarantees are provided to demonstrate that our approach yields MI estimates for the original data. Experiments with high-dimensional data are conducted to highlight the practical advantages of the proposed method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Information plane analysis for dropout neural networks, 2023.
  2. Understanding neural networks and individual neuron importance via information-ordered cumulative ablation. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7842–7852, 2022. doi: 10.1109/TNNLS.2021.3088685.
  3. Entropy estimation via normalizing flow. Proceedings of the AAAI Conference on Artificial Intelligence, 36(9):9990–9998, Jun. 2022. doi: 10.1609/aaai.v36i9.21237. URL https://ojs.aaai.org/index.php/AAAI/article/view/21237.
  4. Training normalizing flows with the information bottleneck for competitive generative classification. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  7828–7840. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/593906af0d138e69f49d251d3e7cbed0-Paper.pdf.
  5. Shannon entropy and mutual information for multivariate skew-elliptical distributions. Scandinavian Journal of Statistics, 40:42–62, 03 2013. doi: 10.1111/j.1467-9469.2011.
  6. Mutual information neural estimation. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.  531–540. PMLR, 07 2018. URL https://proceedings.mlr.press/v80/belghazi18a.html.
  7. Nonparametric independence testing via mutual information. Biometrika, 106, 11 2017. doi: 10.1093/biomet/asz024.
  8. Efficient multivariate entropy estimation via k𝑘kitalic_k-nearest neighbour distances. Ann. Statist., 47(1):288–318, 02 2019. doi: 10.1214/18-AOS1688. URL https://doi.org/10.1214/18-AOS1688.
  9. Flows for simultaneous manifold learning and density estimation. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  442–453. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/051928341be67dcba03f0e04104d9047-Paper.pdf.
  10. Information bottleneck analysis of deep neural networks via lossy compression. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=huGECz8dPp.
  11. High-dimensional dataset entropy estimation via lossy compression. Journal of Communications Technology and Electronics, 66(6):764–768, 7 2021. ISSN 1555-6557. doi: 10.1134/S1064226921060061. URL https://doi.org/10.1134/S1064226921060061.
  12. Gaussianization. In Leen, T., Dietterich, T., and Tresp, V. (eds.), Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000. URL https://proceedings.neurips.cc/paper_files/paper/2000/file/3c947bc2f7ff007b86a9428b74654de5-Paper.pdf.
  13. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp.  2172–2180, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/7c9d0b1f96aebd7b5eca8c3edaa19ebb-Abstract.html.
  14. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006.
  15. Beyond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=25vRtG56YH.
  16. NICE: non-linear independent components estimation. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http://arxiv.org/abs/1410.8516.
  17. Density estimation using real NVP. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HkpbnH9lx.
  18. Normalizing flows for conditional independence testing. Knowledge and Information Systems, 66, 08 2023. doi: 10.1007/s10115-023-01964-w.
  19. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29, 10 2013. doi: 10.1090/jams/852.
  20. MINDE: Mutual information neural diffusion estimation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=0kWd8SJq8d.
  21. Geiger, B. C. On information plane analyses of neural network classifiers—a review. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7039–7051, 12 2022. doi: 10.1109/tnnls.2021.3089037. URL https://doi.org/10.1109%2Ftnnls.2021.3089037.
  22. Estimating information flow in deep neural networks. In ICML, 2019.
  23. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 66(7):4368–4391, 2020. doi: 10.1109/TIT.2020.2975480.
  24. High-dimensional smoothed entropy estimation via dimensionality reduction. In IEEE International Symposium on Information Theory, ISIT 2023, Taipei, Taiwan, June 25-30, 2023, pp.  2613–2618. IEEE, 2023. doi: 10.1109/ISIT54713.2023.10206641. URL https://doi.org/10.1109/ISIT54713.2023.10206641.
  25. Adam: A method for stochastic optimization, 2017.
  26. Glow: Generative flow with invertible 1x1 convolutions. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/2018/file/d139db6a236200b21cc7f752979132d0-Paper.pdf.
  27. On density estimation with diffusion models. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=2LdBqxc1Yv.
  28. Sample estimate of the entropy of a random vector. Problems Inform. Transmission, 23:95–101, 1987.
  29. Formal limitations on the measurement of mutual information. In Chiappa, S. and Calandra, R. (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp.  875–884. PMLR, 08 2020. URL https://proceedings.mlr.press/v108/mcallester20a.html.
  30. On variational bounds of mutual information. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  5171–5180. PMLR, 06 2019. URL https://proceedings.mlr.press/v97/poole19a.html.
  31. Variational inference with normalizing flows. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp.  1530–1538. JMLR.org, 2015. URL http://proceedings.mlr.press/v37/rezende15.html.
  32. Telescoping density-ratio estimation. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  4905–4916. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/33d3b157ddc0896addfb22fa2a519097-Paper.pdf.
  33. Model-powered conditional independence test. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/02f039058bd48307e6f653a2005c9dd2-Paper.pdf.
  34. Opening the black box of deep neural networks via information, 2017.
  35. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1x62TNtDS.
  36. Spivak, M. Calculus On Manifolds: A Modern Approach To Classical Theorems Of Advanced Calculus. Avalon Publishing, 1965. ISBN 9780805390216.
  37. Reasoning About Generalization via Conditional Mutual Information. In Abernethy, J. and Agarwal, S. (eds.), Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp.  3437–3452. PMLR, 09–12 Jul 2020. URL https://proceedings.mlr.press/v125/steinke20a.html.
  38. Invertible autoencoder for domain adaptation. Computation, 7(2), 2019. ISSN 2079-3197. doi: 10.3390/computation7020020. URL https://www.mdpi.com/2079-3197/7/2/20.
  39. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pp.  1–5, 2015.
  40. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pp.  368–377, 1999. URL /brokenurl#citeseer.nj.nec.com/tishby99information.html.
  41. Representation learning with contrastive predictive coding, 2019.
  42. Information-theoretic analysis of generalization capability of learning algorithms. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/ad71c82b22f4f65b9398f76d8be4c615-Paper.pdf.
  43. Diffusion normalizing flow. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  16280–16291. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/876f1f9954de0aa402d91bb988d12cd4-Paper.pdf.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 3 tweets and received 4 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial