What Regularized Auto-Encoders Learn from the Data Generating Distribution (1211.4246v5)

Abstract: What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

• The paper shows that auto-encoders capture the score and Hessian of the log-density, linking reconstruction error to local density gradients.
• The paper provides theoretical generality by proving that its results hold for any auto-encoder parametrization, irrespective of architectural specifics.
• The paper establishes the equivalence between contractive and denoising auto-encoders and demonstrates their utility in approximate MCMC sampling for density estimation.

Overview of "What Regularized Auto-Encoders Learn from the Data Generating Distribution"

The paper presents an in-depth theoretical analysis of regularized auto-encoders, specifically focusing on the information they capture from the data-generating distribution. The authors explore the capacity of these models to characterize local manifold structures in data, offering insights that challenge previous interpretations of reconstruction error as an energy function.

Key Contributions

The primary contributions of the paper include:

1. Score and Hessian Estimation:
   • The paper demonstrates that auto-encoders capture the score, i.e., the derivative of the log-density with respect to the input. This sheds light on their capacity to identify the local gradient of the density, and the work further extends to estimating the Hessian, providing information about the curvature of the log-density.
2. Theoretical Generality:
   • Unlike previous work, the theorems presented apply to any parametrization of the auto-encoder, indicating these results are a fundamental attribute of regularized auto-encoders with sufficient capacity and data.
3. Link between Contractive and Denoising Auto-Encoders:
   • The authors show equivalence between the denoising auto-encoder with small Gaussian corruption and the contractive auto-encoder, where the contraction applies to the entire reconstruction function. This connection supports using these models as alternatives to maximum likelihood methods without the need for a partition function.
4. Sampling Implications:
   • Utilizing the learned score from auto-encoders, the authors propose an approximate Metropolis-Hastings MCMC method for sampling from the estimated distribution. This approach is validated experimentally, showing the auto-encoder’s potential for recovering training set distributions in artificial datasets.

Mathematical Insights

The paper explores mathematical details to illustrate how the optimal reconstruction function captures local density characteristics. By leveraging asymptotic analysis, the work elucidates how the score can be derived from the reconstruction function and the ways in which local means and other moments can be estimated. The results are derived without constraining the auto-encoder to a specific architectural form, demonstrating broad applicability.

Implications and Future Directions

The findings have several implications:

• Theoretical Impact: This work provides a rigorous foundation for understanding the function of reconstruction error in regularized auto-encoders beyond energy models, positioning them as tools for density estimation.
• Practical Utility: By facilitating sampling from complex distributions with auto-encoders, the research suggests potential uses in generative modeling and other AI applications.
• Open Questions: The paper leaves open the problem of extending these principles to discrete data and exploring alternative corruption models, signaling directions for future research.

In conclusion, this research provides a comprehensive theoretical framework for understanding regularized auto-encoders, expanding their role in both theoretical and applied machine learning. The work offers foundational insights that can influence the design of future unsupervised learning algorithms and sampling techniques in artificial intelligence.