Dice Question Streamline Icon: https://streamlinehq.com

Gradient flow for mutual information and algorithmic implementation

Develop a gradient-flow formulation for mutual information itself on the space of probability measures and ascertain whether such a flow can be implemented algorithmically to minimize mutual information.

Information Square Streamline Icon: https://streamlinehq.com

Background

The Fokker–Planck equation is known to be the Wasserstein gradient flow of relative entropy, a perspective that underpins modern analyses of sampling algorithms. The authors suggest an analogous construction for mutual information, which could inspire new dynamics specifically designed to reduce dependence.

An algorithmic realization of a mutual-information gradient flow could offer practical procedures for producing independent or nearly independent samples.

References

While we study the convergence of mutual information along the Langevin diffusion and ULA, many other interesting questions remain open. Finally, just as the Fokker-Planck equation can be viewed as the Wasserstein gradient flow for relative entropy, if we intend to minimize mutual information, then it would be interesting to study the gradient flow for mutual information itself, and whether we can implement it algorithmically.

— Characterizing Dependence of Samples along the Langevin Dynamics and Algorithms via Contraction of $Φ$-Mutual Information (2402.17067 - Liang et al., 26 Feb 2024) in Discussion (Section: Discussion)