Dependency-Aware Mutual Information Gap (DMIG)

• The paper introduces DMIG, a metric that adjusts the Mutual Information Gap by incorporating conditional entropy to account for attribute dependencies.
• It overcomes MIG’s limitations by providing a more realistic disentanglement assessment when semantic attributes are statistically correlated.
• Empirical results, such as those on the NSynth dataset, demonstrate that DMIG better reflects improvements in latent representation quality compared to traditional MIG.

The Dependency-Aware Mutual Information Gap (DMIG) is an evaluation metric for quantifying disentanglement in latent space representations, specifically designed to account for interdependencies among semantic attributes. Developed as a modification of the standard Mutual Information Gap (MIG), DMIG addresses the inadequacy of MIG and similar metrics when attributes are statistically dependent, a scenario frequently encountered in real-world data such as music timbre analysis. By replacing MIG’s independence assumption-based normalization with a conditional entropy term, DMIG provides a more faithful estimate of disentanglement quality in the presence of attribute correlations (Watcharasupat et al., 2021).

1. Foundations: Mutual Information Gap and Its Limitations

The Mutual Information Gap (MIG) was introduced to measure how well individual latent dimensions $z_i$ in representations disentangle known generative factors or semantic attributes $\{a_i\}_{i=1}^M$. In this metric, each latent dimension is ideally associated with one attribute such that $z_i$ captures $a_i$. For discrete attributes, the MIG for attribute $a_i$ is computed as: $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$ where $\mathcal H(a)$ is the Shannon entropy and $\mathcal I(a,b)$ is the mutual information between $a$ and $b$. The denominator assumes that the maximal possible gap between relevant and irrelevant dimensions aligns with attribute entropy, only valid if attributes are statistically independent.

In practical settings, semantic attributes (e.g., brightness and depth in music) are often correlated, violating the independence assumption. Under such correlations, mutual information between an attribute and non-primary latent dimensions ($\{a_i\}_{i=1}^M$0 for $\{a_i\}_{i=1}^M$1) is nonzero, not due to faulty encoding but because of the structural relationship among the attributes themselves. This systematically underestimates the disentanglement as measured by MIG.

2. Dependency-Aware Renormalization: Motivating DMIG

The normalization in MIG, via $\{a_i\}_{i=1}^M$2, acts as an upper bound on the MI gap only under perfect attribute independence. In cases where attributes $\{a_i\}_{i=1}^M$3 and $\{a_i\}_{i=1}^M$4 are correlated, latent dimensions associated with $\{a_i\}_{i=1}^M$5 unavoidably encode information about $\{a_i\}_{i=1}^M$6. Proper evaluation requires allowing for the irreducible information shared due to such dependencies. DMIG renormalizes the MI gap by the conditional entropy $\{a_i\}_{i=1}^M$7, which reflects the residual uncertainty in $\{a_i\}_{i=1}^M$8 given knowledge of its entangled partner $\{a_i\}_{i=1}^M$9. This calibrates the gap measurement so that models are not penalized for encoding unavoidable attribute correlations.

3. DMIG Metric Formulation

For each attribute $z_i$0, let $z_i$1 denote its maximally informative latent dimension, and let $z_i$2 denote the runner-up, defined as $z_i$3. DMIG is then defined as: $z_i$4 When $z_i$5 and $z_i$6 are independent, $z_i$7, so DMIG reduces to MIG. The normalization by conditional entropy directly compensates for the shared information, providing a more accurate measure in the presence of attribute entanglement (Watcharasupat et al., 2021).

4. Computation Procedure

DMIG computation follows these steps:

1. Encoding: Project $z_i$8 samples $z_i$9 through encoder $a_i$0 to obtain $a_i$1.
2. Mutual Information Estimation: For each attribute $a_i$2, estimate $a_i$3 for all $a_i$4.
3. Gap Calculation:
   • Identify primary dimension $a_i$5.
   • Identify runner-up $a_i$6.
   • Compute the MI gap $a_i$7.
4. Normalization:
   • If $a_i$8 (runner-up corresponds to a regularized attribute), compute denominator as $a_i$9.
   • Else, use $a_i$0.
5. DMIG Assignment: $a_i$1 (denominator).

Estimation of MI and (conditional) entropy typically employs discretization or nonparametric strategies such as $a_i$2-nearest neighbors or kernel density estimation, as implemented in standard tools (e.g., scikit-learn, PyInform).

5. Theoretical Properties

• Upper Bound: In cases where $a_i$3 corresponds to a regularized attribute, the maximal possible MI gap equals $a_i$4, ensuring $a_i$5. When $a_i$6, the bound is $a_i$7, consistent with MIG.
• Behavior Under Independence: If $a_i$8, then $a_i$9 and $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$0.
• Continuous Variable Caveat: As differential entropy can be negative, empirical estimates in continuous cases may yield $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$1. This is identified as a limitation and indicates the need for care in entropy estimation or alternative normalization for continuous domains.

6. Empirical Demonstration

On the NSynth dataset, brightness and depth were extracted as correlated attributes using the AudioCommons timbral model. A convolutional VAE ($$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$2) with attribute-regularized $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$3-VAE loss ($$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$4, $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$5) assigned $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$6 to brightness and $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$7 to depth. Three disentanglement measures were tracked:

• Spearman correlation coefficient (SCC) between each attribute and its assigned $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$8
• MIG
• DMIG

Results indicated that SCCs rapidly approached $$\mathrm{MIG}(a_i) = \frac{\mathcal I(a_i, z_i) - \max_{k \neq i} \mathcal I(a_i, z_k)}{\mathcal H(a_i)}$$9, confirming successful encoding. MIG remained near zero throughout training ($\mathcal H(a)$0) due to attribute correlation penalizing the numerator relative to $\mathcal H(a)$1, while DMIG increased toward $\mathcal H(a)$2 and closely followed improvements in SCC. The linear relation between MIG and DMIG demonstrates that DMIG is a renormalized MIG tailored for dependent attributes. Qualitative analysis showed that MIG substantially underreports disentanglement when attributes are correlated, whereas DMIG accurately reflects the model’s performance (Watcharasupat et al., 2021).

7. Practical Considerations and Limitations

DMIG is recommended whenever semantic attributes are plausibly dependent, and functions as a drop-in replacement for MIG. For discrete attributes, entropy and conditional entropy estimation are reliable and DMIG is bounded by $\mathcal H(a)$3. For continuous attributes, negative differential entropy may lead to $\mathcal H(a)$4, motivating the use of capping or alternative denominator adjustments if needed. DMIG retains MIG’s susceptibility to inaccuracies in mutual information estimation; sufficient samples and robust density estimation techniques are crucial. Notably, DMIG does not address all challenges in disentanglement evaluation (e.g., it does not capture multimodality in latents), but corrects the normalization bias that arises from attribute dependencies.

In summary, DMIG preserves MIG’s core focus on the information gap between primary and secondary latent dimensions, while adopting normalization based on conditional entropy to correct for attribute entanglement. This adjustment yields more accurate and interpretable disentanglement metrics in complex, real-world data scenarios where statistical independence among semantic factors cannot be assumed (Watcharasupat et al., 2021).