On Mutual Information Maximization for Representation Learning (1907.13625v2)

Abstract: Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate problems: For example, MI is notoriously hard to estimate, and using it as an objective for representation learning may lead to highly entangled representations due to its invariance under arbitrary invertible transformations. Nevertheless, these methods have been repeatedly shown to excel in practice. In this paper we argue, and provide empirical evidence, that the success of these methods cannot be attributed to the properties of MI alone, and that they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parametrization of the employed MI estimators. Finally, we establish a connection to deep metric learning and argue that this interpretation may be a plausible explanation for the success of the recently introduced methods.

• The paper's main contribution is showing that MI's invariance under invertible transformations limits its ability to learn disentangled representations.
• The study finds that biases from encoder architectures and MI estimator parameterizations largely account for the success of MI-based methods.
• Empirical evaluations highlight that factors like negative sampling and encoder conditioning, akin to metric learning dynamics, are critical for effective performance.

On Mutual Information Maximization for Representation Learning

This paper explores the role of mutual information (MI) in unsupervised or self-supervised representation learning, specifically assessing methods that maximize MI between different views of the data. Although MI-based methods have shown empirical success, the paper argues that MI's properties alone do not account for these results.

Key Insights

1. Challenges with MI Estimation: The paper begins by highlighting the difficulty of estimating MI, particularly in high-dimensional spaces. It also points out that MI's invariance under invertible transformations can lead to entangled representations.
2. Inductive Biases: The authors argue that the success of MI-maximization techniques largely depends on biases introduced by feature extractor architectures and MI estimator parametrizations.
3. Empirical Evaluation: Through several experiments, the paper empirically analyzes the impact of different biases:
   • Bijective Models: Models with bijective encoders can improve downstream classification tasks despite theoretically constant MI.
   • Encoder Conditioning: Maximizing MI estimators may lead toward ill-conditioned encoders, which are hard to invert.
   • Critic Capacity: Higher-capacity critics can provide tighter MI bounds but may yield worse representations compared to simpler critics.
   • Encoder Architecture Influence: Different encoder architectures achieving the same MI lower bounds can result in vastly different performance levels in downstream tasks.
4. Metric Learning Connection: A major part of the paper connects MI maximization with deep metric learning, particularly viewing InfoNCE as a multi-class k-pair loss. This perspective highlights the importance of negative sampling strategies and suggests that recent successes could be attributed more to metric learning dynamics than MI maximization alone.

Theoretical and Practical Implications

The paper raises important points about the theoretical underpinnings and practical execution of MI-based representation learning. It suggests that MI maximization, as traditionally implemented, is not a sufficient criterion for learning effective representations, urging a reevaluation of the role MI plays in these methods.

The paper also points toward future research directions:

• Alternative Metrics: Exploring new measures of information that better capture the structure and constraints of real-world data.
• Co-design of Components: A holistic approach in choosing encoders, critics, and evaluation protocols, potentially leading to better-aligned components for improved performance.
• Beyond Linear Evaluation: Investigating the implications of standard evaluation protocols and how changes could influence representation learning conclusions.

In summary, this work encourages a departure from conventional MI views in unsupervised learning, proposing a blend of insights from deep metric learning and delineating a framework that could guide the development of future representation learning methods.