Importance Weighted Autoencoders (1509.00519v4)

Abstract: The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network's entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.

• The paper proposes a novel methodology that tightens the variational lower bound using multiple importance-weighted samples.
• It employs a Monte Carlo estimator with k-sample importance weighting to improve test log-likelihoods on datasets like MNIST.
• Experiments show that IWAE activates more latent units than VAEs, leading to richer representations and competitive generative performance.

Importance Weighted Autoencoders: A Summary

The paper "Importance Weighted Autoencoders" by Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov addresses a fundamental limitation in the Variational Autoencoder (VAE) framework by proposing an enhancement termed the Importance Weighted Autoencoder (IWAE). Both VAEs and IWAEs operate within the same architectural paradigm, combining a top-down generative network with a bottom-up recognition network. However, the IWAE introduces a stricter and more flexible data log-likelihood lower bound through importance weighting, enabling richer latent space representations and yielding improved test log-likelihood on density estimation tasks.

Background on VAEs

The VAE framework aims to maximize the variational lower bound on the marginal log-likelihood. A recognition network approximates the posterior distribution over latent variables conditioned on the observations, with the objective incentivized by the log-likelihood maximization. VAEs inherently make strong assumptions about the posterior, typically approximating it as factorial and predictable via nonlinear regression. These assumptions can lead to overly simplified representations that do not fully exploit the network's modeling capacity.

IWAE Formulation

The IWAE builds upon the VAE by leveraging a tighter log-likelihood lower bound derived from importance weighting. The primary innovation is using multiple posterior samples to approximate the log-likelihood, offering enhanced flexibility for handling complex posteriors that deviate from the VAE's factorial approximation. The IWAE's training objective utilizes a Monte Carlo estimator based on $k$-sample importance weighting, expressed as: $$L_k(x) = \mathbb{E}_{h_1,\ldots,h_k \sim q(h | x)} \left[ \log \frac{1}{k} \sum_{i=1}^k \frac{p(x,h_i)}{q(h_i|x)} \right].$$ This bound is proven to be tighter than the VAE's bound, with the tightness increasing as the number of samples $k$ grows.

Empirical Evaluation

Experiments conducted on benchmark datasets, MNIST and Omniglot, compare the generative performance of VAEs and IWAEs. The architectures evaluated include models with one or two stochastic layers. It was observed that IWAEs consistently outperformed VAEs, particularly as the number of importance-weighted samples increased. On the MNIST dataset, the IWAE with two stochastic layers achieved a test log-likelihood of -82.90 when $k=50$, which is competitive with state-of-the-art generative models.

A notable observation from the experiments is the tendency of VAEs and IWAEs to use fewer latent dimensions than their capacity allows. The number of active latent units was far below the total number, a phenomenon that did not improve significantly by increasing latent dimensions. However, the IWAE consistently activated more latent units than the VAE, suggesting a richer latent representation.

Implications and Future Directions

The introduction of IWAE addresses a key limitation in the VAE framework, providing a more robust method for modeling complex data distributions. The use of importance weighting to tighten the variational bound represents a significant theoretical advancement in generative modeling. Practically, IWAEs' ability to learn richer latent space representations can enhance various applications, including anomaly detection, semi-supervised learning, and data synthesis.

Future research could explore further improvements in posterior approximation, potentially integrating more sophisticated sampling techniques like normalizing flows or Hamiltonian Monte Carlo. Additionally, extending the IWAE framework to other types of generative models or applying it in conjunction with adversarial training paradigms may yield further advancements in generative performance.

Conclusion

The Importance Weighted Autoencoder significantly enhances the VAE framework by using multiple weighted posterior samples to achieve a tighter log-likelihood bound. This innovation not only improves the generative performance but also enables the learning of more expressive latent space representations. This method represents a meaningful advancement in both theoretical and applied aspects of deep generative modeling.