Stick-Breaking Variational Autoencoders (1605.06197v3)

Abstract: We extend Stochastic Gradient Variational Bayes to perform posterior inference for the weights of Stick-Breaking processes. This development allows us to define a Stick-Breaking Variational Autoencoder (SB-VAE), a Bayesian nonparametric version of the variational autoencoder that has a latent representation with stochastic dimensionality. We experimentally demonstrate that the SB-VAE, and a semi-supervised variant, learn highly discriminative latent representations that often outperform the Gaussian VAE's.

Stick-Breaking Variational Autoencoders

The paper presents a significant advancement in the field of variational autoencoders by introducing a Bayesian nonparametric version termed Stick-Breaking Variational Autoencoder (SB-VAE). The fundamental innovation lies in extending the Stochastic Gradient Variational Bayes (SGVB) technique to accommodate posterior inference for the weights of Stick-Breaking processes, enabling stochastic dimensionality in the latent space.

Theoretical Contributions

The authors address the challenge of using SGVB for nonparametric Bayesian inference where traditional methods are constrained by non-differentiable parametrization issues. This is resolved by employing the Kumaraswamy distribution as an alternative to the Beta distribution. The Kumaraswamy distribution possesses a closed-form inverse CDF, which allows SGVB to be applied effectively within this framework. This builds upon the need for a differentiable non-centered parametrization.

Improved Model: SB-VAE

The introduction of SB-VAE marks a departure from the conventional Gaussian assumptions in variational autoencoders. By leveraging stick-breaking processes, SB-VAE provides an infinite dimensional latent representation that can adapt its capacity depending on data complexity. This self-determined width enables better generative modeling with improved latent representations.

Empirical Results

The experiments conducted demonstrate that SB-VAE, and its semi-supervised variant, exhibit superior discriminative properties in their latent representations compared to Gaussian VAEs. The authors present empirical results on image datasets such as MNIST and SVHN, highlighting better performance in both unsupervised and semi-supervised learning tasks.

1. Discriminative Qualities: SB-VAE showed enhanced class boundary preservation, which was assessed using k-Nearest Neighbors classifiers on latent representations. The lower error rates indicate better class discrimination in the latent space.
2. Adaptability: The SB-VAE's ability to adaptively increase latent dimensionality for complex data (e.g., rotated digits in MNIST+rot) was demonstrated, showcasing its data-dependent flexibility.

Implications and Future Work

The integration of Bayesian nonparametrics within SGVB frameworks represents a notable progress in scalability for deep generative models. The potential applications extend beyond variational autoencoders, offering prospects for more dynamically adaptive neural network architectures. The future research directions might explore full Dirichlet processes with non-trivial base measures and their integration in more complex neural networks.

Conclusion

Stick-Breaking Variational Autoencoders present a promising avenue in deep generative modeling by enhancing model flexibility and providing stronger latent representations. The investigative use of the Kumaraswamy distribution to support SGVB in nonparametric settings exemplifies the paper's methodological rigor and the potential for further innovations in probabilistically principled neural network design.