Glow: Generative Flow with Invertible 1x1 Convolutions (1807.03039v2)

Abstract: Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and parallelizability of both training and synthesis. In this paper we propose Glow, a simple type of generative flow using an invertible 1x1 convolution. Using our method we demonstrate a significant improvement in log-likelihood on standard benchmarks. Perhaps most strikingly, we demonstrate that a generative model optimized towards the plain log-likelihood objective is capable of efficient realistic-looking synthesis and manipulation of large images. The code for our model is available at https://github.com/openai/glow

• The paper introduces Glow, a flow-based generative model that leverages learnable invertible 1x1 convolutions to enhance expressivity and training efficiency.
• It integrates actnorm and affine coupling layers for reversible transformations, enabling exact log-likelihood computation and efficient inference.
• Experimental results on benchmarks like CIFAR-10 and ImageNet show that Glow outperforms previous models with lower bits per dimension and higher image quality.

Glow: Generative Flow with Invertible 1x1 Convolutions

The paper "Glow: Generative Flow with Invertible 1x1 Convolutions," authored by Diederik P. Kingma and Prafulla Dhariwal from OpenAI, presents a new type of flow-based generative model referred to as Glow. This model introduces the concept of an invertible $1 \times 1$ convolution, which improves both the training efficiency and the quality of the generated samples. This paper is a notable contribution to the field of generative modeling, especially in the context of scalable and efficient models.

Background and Motivation

Generative modeling, particularly likelihood-based generative models, has seen substantial advances in recent years. The appeal of these models lies in their tractability for exact log-likelihood evaluation, the ability to infer exact latent variables, and the feasibility of parallelized training and synthesis. Flow-based generative models, such as NICE and RealNVP, have demonstrated capabilities that set them apart from other likelihood-based models like autoregressive models and Variational Autoencoders (VAEs).

Existing methods like GANs, despite their prowess in image synthesis, face challenges related to latent-space encodings, full data support, and optimization difficulties. On the other hand, flow-based models offer exact inference, efficient synthesis, and potential for downstream tasks due to their invertible and parallelizable structures. However, they have historically received less attention relative to GANs and VAEs.

Contributions

The primary contribution of this paper is the introduction of Glow, a flow-based model that replaces the fixed permutation of channels used in previous works with a learned invertible $1 \times 1$ convolution. This alteration is not trivial, as it aims to enhance the expressivity and efficiency of the model. Key features of Glow include:

1. Actnorm Layers: These scale and translate activations using trainable parameters that are initialized to normalize the activations based on the initial data batch, thereby facilitating improved training dynamics.
2. Invertible $1 \times 1$ Convolution: This component generalizes the permutation operation to allow for learned transformations, making the model more flexible and expressive, while maintaining efficiency.
3. Affine Coupling Layers: Inspired by prior works such as NICE and RealNVP, these layers allow for efficient reversible transformations, which are essential for the exact computation of log-likelihood.

Methodology

The Glow model builds upon the conceptual framework of NICE and RealNVP, integrating an invertible $1 \times 1$ convolution within the flow steps that consist of actnorm, convolution, and coupling layers. This structure is embedded within a multi-scale architecture to enable effective hierarchical representation learning.

The $1 \times 1$ convolution is particularly notable for its initialization as a random rotation matrix, ensuring efficient determinant computation. By parameterizing this transformation via LU decomposition, the model achieves computational efficiency, significantly reducing the cost of determinant calculations for large channel sizes.

Experimental Results

Quantitative experiments comparing Glow with RealNVP demonstrate substantial improvements across various benchmarks:

• CIFAR-10: Achieved a negative log-likelihood (bits per dimension) of 3.35, compared to 3.49 by RealNVP.
• ImageNet 32x32: Improved to 4.09 from RealNVP's 4.28.
• ImageNet 64x64: Enhanced results showing 3.81 bits per dimension as opposed to 3.98.
• LSUN: Across different subsets like Bedroom, Tower, and Church Outdoor, Glow outperformed RealNVP significantly.

Qualitative experiments on the CelebA-HQ dataset further validate Glow's effectiveness in generating high-resolution, realistic images and its ability to perform meaningful interpolations and semantic manipulations in the latent space. These capabilities are particularly illustrated by the smooth image transitions through latent space interpolations and the targeted modification of image features (e.g., inducing a smile or changing hair color).

Implications and Future Directions

Glow presents a significant step forward for flow-based generative models. Its advancements can impact various practical applications such as image synthesis, semi-supervised learning, and model-based control. The model's efficiency in both training and inference underscores its potential for deployment in real-time applications.

Future research could extend Glow to more complex data modalities beyond images, such as audio or video, and explore further optimizations to scale the models to higher resolutions or more intricate datasets. Additionally, incorporating mechanisms for controlling and enhancing the diversity of generated samples remains an open field for investigation.

In summary, Glow's introduction of invertible $1 \times 1$ convolutions represents a substantial enhancement in the landscape of generative modeling, providing a compelling balance of expressivity, efficiency, and robustness. The model demonstrates prowess not only in quantitative evaluations but also in qualitative synthesis tasks, promising further exciting developments in the field of generative flow-based models.