Mutual Information Proxies

• Mutual information proxies are alternative functionals designed to approximate, bound, or generalize Shannon MI using varied mathematical forms tailored to computational and statistical challenges.
• They include families like Rényi-type generalizations, low-dimensional projections, variational bounds, and neural estimators, each suited for specific tasks in communication, feature selection, and learning.
• These proxies provide tractable, robust MI estimation in high-dimensional applications while addressing key issues such as bias–variance trade-offs and model mismatch.

Mutual information proxies are alternative functionals or algorithmic surrogates designed to approximate, bound, or generalize the Shannon mutual information (MI) between random variables. These proxies play a crucial role in high-dimensional inference, empirical estimation, information-theoretic learning, and communication theory, where the exact computation or optimization of MI is computationally expensive, statistically unstable, or otherwise infeasible. A diverse ecosystem of such proxies exists, ranging from variational bounds and statistical surrogates to Rényi-type generalizations, low-dimensional or pairwise approximations, and algorithmic relaxations tuned for specific operational problems.

1. Structural Families of Mutual Information Proxies

Several distinct mathematical forms serve as mutual information proxies:

• Rényi-Type Generalizations: α-mutual information family, parameterized by the Rényi order α, includes Sibson, Arimoto, Augustin–Csiszár, and Lapidoth–Pfister α-MI. All converge to Shannon MI as α → 1. Each defines a convex divergence or entropy-based relaxation, enabling operational characterizations tuned to heavy-tailed, rare-event, or adversarial settings (Kamatsuka et al., 2024).
• Marginal/Adjacency-Structure Proxies: Factorized multi-information (FMI) and separated factorized mutual information (SFMI) average multi-information or MI over selected marginals, greatly reducing estimation and computational complexity while retaining sharp characterizations—e.g., full mutual information maximizers are preserved in optimizers under connected margin collections (Merkh et al., 2019).
• Low-Dimensional/Projection Proxies: Max-sliced mutual information (mSMI) restricts attention to the largest MI obtainable via k-dimensional linear projections, interpolating between classical CCA and Shannon MI for tractable higher-order dependency capture in high-dimensional data (Tsur et al., 2023).
• Statistical/Empirical Bounds: Proxies based solely on adjacency relations or marginal-event statistics provide computable upper and lower MI bounds in discrete or constrained channels, crucial for deletion channels and similar models where full entropy calculations are hard (Han et al., 2015).
• Alternative Divergence/Metric Surrogates: Jensen–Shannon, total variation, Wasserstein (optimal transport), and maximum mean discrepancy (MMD) divergences between joint and product-of-marginals define proxies that retain some operational informativeness, and in well-behaved cases coincide with MI maximizers or saturate theoretical bounds (Kuskonmaz et al., 2022).
• Variational and Neural Estimation Proxies: Variational lower bounds (Donsker–Varadhan (DV), InfoNCE, NWJ), as well as neural estimators (MINE, SMILE, MMG diffusion-based), yield scalable, statistically efficient surrogates for mutual information in high-dimensions, with diagnostic procedures for bias, variance, and overfitting (Abdelaleem et al., 31 May 2025, Czyż et al., 2023, Yu et al., 24 Sep 2025).
• Compression and Rate-Distortion Proxies: In single-shot or non-asymptotic compression, MI minimization under distortion/fidelity constraints is within additive O(log R) bits of the actual entropy-optimal encoding, providing a convex proxy for practical code design (Kostina, 7 Feb 2026).

2. Rényi-Type α-Mutual Information Proxies

A central structural generalization is the α-parametrized Rényi mutual information, with four main forms defined for discrete distributions (Kamatsuka et al., 2024):

Proxy	Definition	Key Variational Form
Sibson’s α-MI	$$I_\alpha^{(S)}(X;Y) = \min_{q_Y} D_\alpha(p_X p_{Y|X} \| p_X q_Y)$$	Single-minimization (closed form for q_Y)
Arimoto’s α-MI	$$I_\alpha^{(A)}(X;Y) = H_\alpha(X) - H_\alpha^{(A)}(X|Y)$$	Closed via E₀ function
Augustin–Csiszár’s α-MI	$$I_\alpha^{(C)}(X;Y) = \min_{q_Y} \mathbb{E}_{X}[D_\alpha(p_{Y|X}(\cdot|X) \| q_Y)]$$	Saddle-point/min-max over channel and marginal
Lapidoth–Pfister’s α-MI	$$I_\alpha^{(LP)}(X;Y) = \min_{q_X, q_Y} D_\alpha(p_X p_{Y|X} \| q_X q_Y)$$	Joint minimization

All coincide with the Shannon MI at $\alpha=1$ and maintain monotonicity properties with respect to independence. These proxies are crucial for operational problems in coding, privacy, and robust estimation, and have efficient alternating-optimization (AO) schemes—often generalizing the Arimoto–Blahut approach—facilitating practical optimization over auxiliary distributions or channel reverse maps.

3. Variational, Neural, and Estimation-Based Proxies

Proxies derived from variational principles, neural-critic architectures, and training-time regularization address the practical estimation of MI in data-driven settings (Abdelaleem et al., 31 May 2025, Czyż et al., 2023):

• DV/Barber–Agakov Lower Bounds: For any scoring function or critic $T$, $$I(X;Y) \geq \mathbb{E}_{p_{X,Y}}[T(X,Y)] - \log \mathbb{E}_{p_X p_Y}[e^{T(X,Y)}]$$, attaining equality when $T$ matches the pointwise MI up to constant shift (Choi et al., 2023).
• InfoNCE and Contrastive Approaches: Surrogates of MI based on in-batch negative sampling, yielding efficient lower bounds that are matchable to the true MI as batch size grows, with exact tightness when the critic equals the pointwise MI plus data-dependent bias.
• MMG (Diffusion-Based) Proxies: Mutual information expressed exactly as a half-integral of the gap between unconditional and conditional MMSEs across all SNRs in denoising diffusion models. This admits robust neural-network-based estimation that passes high-dimensional consistency tests and establishes connections with minimum mean square error theory (Yu et al., 24 Sep 2025).
• Shrinkage and Bayesian Nonparametrics: Empirical Bayes shrinkage and Bayesian nonparametric (Dirichlet process) smoothing provide range-preserving, robust MI surrogates with established minimax mean-squared-error properties, especially suitable for small-sample or sparse contingency computations (Sechidis et al., 2016, Al-Labadi et al., 2021).

4. Proxy Design in Feature Ranking, Data Compression, and Communication

Application-driven proxies select functional forms suited to optimization, interpretability, or code design:

• Feature Selection/Biomarker Ranking: Chain-rule expansions for MI (e.g., only first- and second-order terms) serve as proxies for conditional mutual information in greedy ranking objectives. These drastically reduce data and computation requirements, with empirical Bayes estimators controlling mean-square error in high-dimensional, undersampled regimes (Sechidis et al., 2016).
• Single-Shot Compression: Mutual information minimization under pointwise/average distortion or fidelity constraints bounds the entropy-optimal description length of the source within a logarithmic gap, making MI a sound proxy in variable-length, non-asymptotic source coding (Kostina, 7 Feb 2026).
• Action and Adjacency-Based Bounds: In discrete channels, explicit lower and upper MI bounds arise from adjacency relations or random action distributions over channel mappings; these are computationally accessible even when the joint distribution is intractable, and offer strict improvements over classical codebook-based bounds in communication theory (Han et al., 2015).

5. Algorithmic and Statistical Properties, Complexity, and Convergence

MI proxies differ in statistical robustness, sample complexity, and computability:

• AO Convergence for α-MI: For all α-proxies, AO algorithms—single-step for Sibson, explicit for Arimoto, nested or block-alternate for Augustin–Csiszár and Lapidoth–Pfister—yield global convergence to the proxy value, with provable O(1/k) error decay in Csiszár’s case, and fastest convergence in cases with closed-form updates (Kamatsuka et al., 2024).
• Low-Dimensional/Pairwise Proxies: FMI, SFMI, and similar factorized surrogates reduce storage/estimation from exponential to polynomial in dimension, and in the connected case, maintain optimizer sets identical to the original multi-information, with exact maximizers preserved.
• Projection and Margin-Based Tightness: Max-sliced MI, margin-based surrogates, and geometric decompositions (e.g., layerwise probing) deliver sharp proxies when dependence is low-dimensional or linearly/separably embedded, with concrete non-asymptotic error rates and explicit geometric error decay proportional to margin width (Tsur et al., 2023, Choi et al., 2023).
• Variational and Neural Estimator Diagnostics: Reliable estimation requires regularization (early stopping, subsampling, extrapolation, smoothing) and verification of linear extrapolation in data subsamples; neural estimators such as InfoNCE and SMILE-VSIB achieve unbiased MI surrogates only when the critic is expressively sufficient and data latent structure is low-dimensional (Abdelaleem et al., 31 May 2025, Czyż et al., 2023).

6. Theoretical Limits, Operational Implications, and Use Cases

• Fisher Information as MI Upper/Lower Bound: In both classical and quantum settings, Fisher information bounds on MI provide operational proxies for estimation and communication capacity, with easily computable (reparametrization-invariant) closed forms extending to scenarios (e.g., finite, sharp priors) where van Trees or Efroimovich-type bounds become trivial (Barnes et al., 2021, Górecki et al., 2024).
• Proxy Suitability and Limiting Factors: No proxy is uniformly unbiased for all data distributions at finite sample size; proxy selection is task- and data-dependent, grounded in theoretical properties such as invariance, minimax optimality, or analytical tractability.
• Applications: Proxies underpin discrimination, dependence detection, feature selection, fairness constraint relaxation, representation learning, lossy compression, and adaptive channel coding, with empirical evidence demonstrating efficacy in computer vision, speech, communication, and biomarker discovery domains (Tsur et al., 2023, Kostina, 7 Feb 2026, Abdelaleem et al., 31 May 2025).

7. Limitations, Open Problems, and Future Directions

Despite their practical and theoretical utility, all proxies face limitations:

• Estimation in high latent dimension remains infeasible for most neural and k-NN estimators due to the curse of dimensionality (Abdelaleem et al., 31 May 2025).
• Bias–variance trade-offs, especially in variational or shrinkage-based methods, are sensitive to architectural and smoothing hyperparameters (Czyż et al., 2023, Sechidis et al., 2016).
• Proxy tightness under model mismatch, modality structure, or non-standard dependence is an active area of analytical and empirical research.
• Developing robust, model-agnostic diagnostics for proxy reliability, as well as automatic proxy selector frameworks, remains an open challenge.

Ongoing research seeks to extend proxy frameworks to conditional MI, joint information measures, more general divergence classes, and operational tasks in robust statistics, privacy, and adversarial learning, further entrenching proxy functionals as central tools in modern information-theoretic methodology.

---

References:

• Max-Sliced Mutual Information (Tsur et al., 2023)
• Mutual Information Bounds via Adjacency Events (Han et al., 2015)
• Accurate Estimation of Mutual Information in High Dimensional Data (Abdelaleem et al., 31 May 2025)
• Ranking Biomarkers Through Mutual Information (Sechidis et al., 2016)
• Factorized Mutual Information Maximization (Merkh et al., 2019)
• MMG: Mutual Information Estimation via the MMSE Gap in Diffusion (Yu et al., 24 Sep 2025)
• On the Properties and Estimation of Pointwise Mutual Information Profiles (Czyż et al., 2023)
• MIM: Mutual Information Machine (Livne et al., 2019)
• Understanding Probe Behaviors through Variational Bounds of Mutual Information (Choi et al., 2023)
• Fisher Information and Mutual Information Constraints (Barnes et al., 2021)
• A Bayesian Nonparametric Estimation of Mutual Information (Al-Labadi et al., 2021)
• Mutual Information Bounded by Fisher Information (Górecki et al., 2024)
• Investigation of Alternative Measures for Mutual Information (Kuskonmaz et al., 2022)
• A Bayesian Framework for Information-Theoretic Probing (Pimentel et al., 2021)
• Single-shot lossy compression: mutual information bounds (Kostina, 7 Feb 2026)
• Alternating Optimization Approach for Computing $\alpha$-Mutual Information and $\alpha$-Capacity (Kamatsuka et al., 2024)