Hyperspherical Variational Auto-Encoders (1804.00891v3)

Abstract: The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models. But although the default choice of a Gaussian distribution for both the prior and posterior represents a mathematically convenient distribution often leading to competitive results, we show that this parameterization fails to model data with a latent hyperspherical structure. To address this issue we propose using a von Mises-Fisher (vMF) distribution instead, leading to a hyperspherical latent space. Through a series of experiments we show how such a hyperspherical VAE, or $\mathcal{S}$-VAE, is more suitable for capturing data with a hyperspherical latent structure, while outperforming a normal, $\mathcal{N}$-VAE, in low dimensions on other data types. Code at http://github.com/nicola-decao/s-vae-tf and https://github.com/nicola-decao/s-vae-pytorch

• The paper presents the S-VAE model, replacing the Gaussian prior with a von Mises-Fisher distribution to better capture the hyperspherical structure of directional data.
• It innovatively extends the reparameterization trick using rejection sampling to effectively optimize the vMF concentration parameter and KL divergence.
• Experiments on datasets like MNIST and citation networks demonstrate improved reconstruction errors and superior link prediction compared to standard VAE models.

Analysis of "Hyperspherical Variational Auto-Encoders"

The paper "Hyperspherical Variational Auto-Encoders" by Davidson et al. presents an innovative approach aimed at enhancing the adaptability of the Variational Auto-Encoder (VAE) framework, particularly with data exhibiting a hyperspherical latent structure. The authors propose replacing the traditional Gaussian distribution with a von Mises-Fisher (vMF) distribution, which results in a hyperspherical latent space. This approach endeavors to provide a more appropriate fit for directionally distributed data often found in real-world applications.

Key Contributions

The key contribution is the introduction of the $\mathcal{S}$-VAE, a model utilizing a vMF distribution for both prior and posterior approximations. Traditional $\mathcal{N}$-VAEs use Gaussian distributions that, while mathematically convenient, can be suboptimal for data naturally aligned on spherical manifolds. The proposed $\mathcal{S}$-VAE addresses these limitations by fitting the model to the spherical nature of data, like directional datasets, thereby ensuring more robust learning in low-dimensional spaces.

Methodology

The methodology involves substituting the Gaussian distribution with a vMF distribution. The vMF is analogous to the Gaussian distribution but defined on the hypersphere. The authors extend the reparameterization trick to handle rejection sampling, thus enabling the learning of the vMF distribution's concentration parameter ($\kappa$).

The paper discusses the benefits of vMF distribution in terms of KL divergence optimization. Unlike Gaussian assumptions that might tighten probabilities around the origin in low dimensions, vMF supports a uniform distribution over the hypersphere, beneficial for hyperspherical latent spaces.

Numerical Results and Experiments

The experimental results demonstrate notable improvements using the $\mathcal{S}$-VAE across various settings:

1. Synthetic Data and MNIST: The $\mathcal{S}$-VAE effectively recovered hyperspherical latent structures, outperforming $\mathcal{N}$-VAEs in preserving the data's intrinsic geometry. On the MNIST dataset, $\mathcal{S}$-VAE showed improved negative reconstruction error performance in lower dimensions.
2. Semi-Supervised Learning: In semi-supervised tasks, the $\mathcal{S}$-VAE provided better class separation, outperforming standard VAEs by utilizing the entire latent space effectively due to its uniform prior.
3. Link Prediction: The $\mathcal{S}$-VAE demonstrated efficacy in citation network datasets. On the Cora and Citeseer datasets, it showed superior performance in link prediction tasks compared to $\mathcal{N}$-VGAEs, likely due to the appropriate embedding space choice.

Future Implications and Theoretical Considerations

The inclusion of hyperspherical representations introduces a new dimension to unsupervised learning models. Future work could explore integrating this framework with flexible posterior distributions via normalizing flows on hyperspheres or a combination of hyperspherical and hyperplanar spaces.

The research also opens pathways for examining how the topology of latent spaces influences the underlying representation learning. Given the initial promising results, expanding these methods to higher dimensions could unveil more complexities related to surface area collapse and curse of dimensionality in hyperspherical spaces.

Conclusion

The introduction of hyperspherical latent spaces through $\mathcal{S}$-VAEs represents a pivotal modification to conventional VAE methodologies, aligning model assumptions with data structures inherently present in various application domains. The paper offers robust theoretical foundations and empirical evidence, highlighting the benefits of reshaping latent space assumptions based on domain-specific data characteristics. This work is well-poised to inspire further exploration into manifold learning and applications in directionally structured data.

Overall, this paper contributes significantly to the field of unsupervised learning by challenging standard Gaussian distribution usage, urging a reconsideration of latent space manifold structures in autoencoder designs.