Entropy and mutual information in models of deep neural networks (1805.09785v2)

Abstract: We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics methods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method. (iii) We propose an experiment framework with generative models of synthetic datasets, on which we train deep neural networks with a weight constraint designed so that the assumption in (i) is verified during learning. We study the behavior of entropies and mutual informations throughout learning and conclude that, in the proposed setting, the relationship between compression and generalization remains elusive.

• The paper introduces a heuristic method from statistical physics to compute entropies and mutual information in deep neural networks.
• It offers rigorous analytical proofs under conditions like Gaussian weights and two-layer architectures, clarifying asymptotic network behavior.
• Experiments on synthetic data reveal that while bounded activations show entropy compression, the expected link to enhanced generalization is not confirmed.

Insights from "Entropy and mutual information in models of deep neural networks"

The paper "Entropy and mutual information in models of deep neural networks" explores a refined information-theoretical analysis of deep learning models by computing entropies and mutual information for a class of neural networks. The paper is motivated by the desire to empirically validate or disprove the intuition that compression within hidden representations of deep networks should correlate with improved generalization performance.

Key Contributions:

1. Heuristic Derivation and Computational Method:
   • The paper presents a method for deriving entropies and mutual information from statistical physics principles. By assuming weight matrices are independent and orthogonally invariant, the authors suggest that this approach allows for a computationally feasible way to analyze deep network behavior.
   • A specific formula is provided to evaluate information-theoretic quantities, facilitating the analysis even in high-dimensional, non-linear feed-forward neural networks trained on synthetic datasets.
2. Rigorous Analytical Results:
   • Under certain conditions, including Gaussian weights and specific network architectures like two-layer networks, the authors provide a proof using the adaptive interpolation method. This proof advances the understanding of how these models operate under asymptotic conditions, legitimizing the use of their heuristic methods for broader cases.
3. Empirical Framework and Evaluation:
   • The authors introduce a practical framework where they experiment on generative models with synthetic data, analyzing trained neural networks. This empirical setting adheres to the constraints needed by their theoretical model, validating the applicability of their computed results in practice.
   • In this controlled setting, mutual information measurements point out the lack of persistence of the hypothesized link between compression of neural representations and improved generalization.

Numerical Experiments and Findings:

The researchers investigated multiple scenarios including a teacher-student setup and settings mimicking variational autoencoder structures. By conducting various experiments, they examined differences in information dynamics across several activation functions in neural networks and found that:

• Networks with bounded activations, such as hardtanh, exhibit entropy compression due to saturation effects. However, unbounded activations, like ReLU, did not show a similar reduction.
• In large neural networks with non-linear activations, mutual information often decreased during training—a phenomenon sometimes seen as supporting the intuitive compression-generalization hypothesis, though clear causation remains undemonstrated.

An additional key finding is the sensitivity of mutual information dynamics to initialization and weight scaling factors, suggesting this "compression" mostly mirrors complexities in the optimization landscape rather than simply arising from fixed architectural properties of neural networks.

Implications for Future AI Development:

This investigation implies that the interpretation of neural compression as a general principle for regularization is not straightforward and results in the need for caution. The proposed method demonstrates a potential avenue for deeper analytical interpretations of neural networks without simplifying the structure excessively, thus offering a more sophisticated toolkit for disentangling the aspects of training responsible for network generalization. Future work might explore further extensions of the presented formula to cover neural network architectures beyond the orthogonally invariant matrices and include biases effectively. Additionally, the adaption of this methodology could provide insights into unsupervised learning problems and more complex architectures like transformers or graph neural networks, hence broadening the horizons of applicability.

In summary, while the research provides evidence of complexity compression not translating into clear practical guidelines for improved generalization, it enhances the information-theoretic understanding of network dynamics—laying groundwork for more rigorous AI system analyses.