From Variational to Deterministic Autoencoders (1903.12436v4)

Abstract: Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of VAEs. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without forcing it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data, we introduce an ex-post density estimation step that can be readily applied also to existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules. \footnote{An implementation is available at: \url{https://github.com/ParthaEth/Regularized_autoencoders-RAE-}}

• The paper presents Regularized Autoencoders (RAEs) as a deterministic alternative to VAEs, addressing latent space and quality issues.
• The paper demonstrates that employing novel regularization schemes and an ex-post density estimation step yields improved generative performance.
• The paper highlights a simplified training process with reduced hyperparameter sensitivity, suggesting broader applications in generative modeling.

From Variational to Deterministic Autoencoders: An Evaluation

The paper explores an innovative approach toward generative modeling, primarily addressing the limitations inherent in Variational Autoencoders (VAEs). While VAEs have become a cornerstone for deep generative models due to their theoretical grounding and potential, they also present challenges relating to model complexity, latent space structure, and sample quality. In this work, the authors propose a transition from a variational paradigm to a deterministic one, leading to the development of Regularized Autoencoders (RAEs).

The authors start by critiquing the VAE framework, highlighting its tendency to balance sample quality and reconstruction quality inadequately due to over-simplified prior distributions or the over-regularization from the Kullback-Leibler (KL) divergence term in the VAE objective. VAEs require approximating expectations that lead to increased gradient variance, making training more dependent on the careful selection of hyperparameters. Often, trained VAEs display a mismatch between the aggregated posterior and the presumed prior, affecting the downstream sample quality adversely.

By reinterpreting the stochastic encoding process as injecting Gaussian noise into a deterministic decoder, the authors introduce RAEs that replace this noise injection with alternative regularization schemes. These regularization methods aim to maintain smooth latent spaces without imposing simplistic distributions. Notably, the authors employ an ex-post density estimation step to account for producing novel samples, which can be extended to existing VAEs, thus improving the sample quality.

RAEs exhibit a simplified architecture and training process relative to VAEs while retaining competitive sample generation capabilities. Different regularization schemes, such as Tikhonov regularization, gradient penalties, and spectral normalization, were explored, with empirical studies demonstrating RAEs' proficiency in generating comparable or superior outputs to VAEs in several tasks, including image and molecular data generation.

Empirical results show that RAEs, when equipped with density estimation mechanisms, achieve high-quality generative capability. For instance, RAEs frequently outperform VAEs in standard image datasets such as MNIST, CIFAR-10, and CelebA when the models are assessed using Fréchet Inception Distance (FID). These performance metrics signify that RAEs can effectively smooth and structure the latent space without the classical KL-divergence-induced regularization.

The implications of this work are broad. Practical applications benefit from the reduced complexity and improved sample quality RAEs offer. Theoretically, this research encourages rethinking the deterministic potential in generative models, suggesting avenues for further inquiry into architectures that could reduce reliance on stochastic approximations.

Given these advancements, one might speculate that future work could focus on refining RAEs' regularization schemes or further dissecting the role of latent space structures in generative modeling. Additionally, cross-domain applications could harness these deterministic frameworks, extending beyond the domain of visual data to other rich data structures in areas such as natural language processing or audio synthesis.

In conclusion, this paper paves the way for a better understanding of how generative models can be effectively regularized and structured without the conventional variability constraints imposed by VAEs. The exploration of deterministic frameworks, as evidenced by the success of RAEs, could herald new directions in the development and optimization of deep generative models.