---
title: 'Glow: Invertible Flow-Based Model'
url: https://www.emergentmind.com/topics/glow-b89d0b18-8cb8-408c-a2ec-489f2a9e65fa
type: topic
---

# Glow: Invertible Flow-Based Model

Glow usually denotes the flow-based generative model introduced in "Glow: Generative Flow with Invertible 1x1 Convolutions" [1807.03039]. It defines an invertible mapping $f_1 \circ f_2 \circ \cdots \circ f_K$ from data $x \in \mathbb{R}^{h \times w \times c}$ to latent $z$, making exact density evaluation, exact latent-variable inference, and parallelizable training and synthesis simultaneously tractable. Within the class of flow-based generative models, Glow is characterized by a simple three-part flow step—ActNorm, an invertible $1\times1$ convolution, and an affine coupling layer—combined with a multi-scale architecture and implementation choices aimed at stabilizing very deep, high-resolution models [1807.03039].

## 1. Formal definition and probabilistic structure

Glow models the data distribution by an invertible transformation from image space to a latent space with a simple tractable prior, such as $\mathcal{N}(0,I)$. Exact log-density follows from the change-of-variables formula:
$$
\log p_X(x) = \log p_Z(z) + \sum_{i=1}^K \log \left| \det \frac{\partial f_i(h_{i-1})}{\partial h_{i-1}} \right|.
$$
Here $h_0=x$, the intermediate states are produced by the sequence of invertible transformations, and $z$ is the final latent representation [1807.03039].

This formulation makes the model analytically different from approximate-likelihood or implicit generators: density evaluation is exact, latent inference is exact because the mapping is invertible, and synthesis is obtained by sampling from the prior and applying the inverse transformation. The architecture is therefore organized so that every constituent operation has an inexpensive inverse and a tractable log-Jacobian determinant.

## 2. Invertible building blocks of a Glow step

Each Glow flow step interleaves three inexpensive invertible operations [1807.03039].

**ActNorm.** Glow applies a per-channel scale $s \in \mathbb{R}^c$ and bias $b \in \mathbb{R}^c$:
$$
\forall i,j:\quad y_{i,j}=s \odot x_{i,j}+b.
$$
The inverse is $x=(y-b)/s$, and the log-Jacobian is
$$
\log \left| \det \frac{\partial y}{\partial x} \right| = h \cdot w \cdot \sum_c \log |s_c|.
$$
Unlike batch-norm, ActNorm is initialized on a single minibatch to make each channel zero-mean and unit-variance, after which $s$ and $b$ are trained as free parameters. This is used to stabilize very deep flows even with one-image batches per GPU.

**Invertible $1\times1$ convolution.** Glow replaces fixed channel permutations with a learned dense channel-mixing transform:
$$
\forall i,j:\quad y_{i,j}=W x_{i,j},
$$
where $W \in \mathbb{R}^{c\times c}$. The inverse is $x=W^{-1}y$, and the log-Jacobian is
$$
h \cdot w \cdot \log |\det W|.
$$
Because computing $\det W$ is $O(c^3)$, Glow optionally parameterizes $W$ by an LU decomposition,
$$
W = P \cdot L \cdot (U + \operatorname{diag}(s)),\qquad \log |\det W|=\sum_c \log |s_c|,
$$
reducing the per-forward-pass cost from $O(c^3)$ to $O(c)$.

**Affine coupling layer.** The channels are split as $x \to [x_a,x_b]$. A small convolutional network computes $(\log s,t)=NN(x_b)$, and the transformation is
$$
y_a=s \odot x_a+t,\qquad y_b=x_b,\qquad y=[y_a,y_b].
$$
Inversion is trivial,
$$
x_a=(y_a-t)/s,\qquad x_b=y_b,
$$
and the Jacobian contribution is $\sum_c \log |s_c|$. Glow uses zero initialization on the last convolution in this network so that each coupling layer starts as the identity, easing optimization of very deep models.

## 3. Channel mixing, hierarchy, and multi-scale factorization

The invertible $1\times1$ convolution is the architectural feature that most sharply distinguishes Glow from earlier fixed-permutation flow designs. Learning a dense $W$, initialized to a random rotation, substantially increases mixing across channels compared to fixed or random permutations, and this yields faster convergence and better final likelihoods [1807.03039].

Glow also introduces a multi-scale hierarchy. It stacks $K$ flow steps per level and has $L$ levels. After each level, half of the channels are “squeezed” out and factored into the prior, while the remainder are passed to the next level. This splitting reduces memory growth and encourages hierarchical latents at multiple spatial resolutions. The resulting latent representation is therefore distributed across scales rather than concentrated in a single terminal tensor.

The reported qualitative behavior is consistent with this design. Deeper models, specifically $L=6$ versus $4$, capture longer-range coherence. This suggests that Glow’s hierarchy is not only a memory-management device but also a mechanism for progressively modeling coarse-to-fine image structure.

## 4. Optimization and implementation regime

Glow is trained by maximum likelihood on dequantized data of the form $(x+u,\; u \sim \mathrm{Uniform}[0,1/a])$ using Adam with learning rate $10^{-3}$, $\beta_1=0.9$, and $\beta_2=0.999$ [1807.03039]. Typical minibatch sizes are $512$ for CIFAR-10, $128$ for ImageNet32, and as low as $1$ image per GPU at $256\times256$ resolution.

Several engineering choices are integral to the training recipe. ActNorm replaces batch-dependent normalization and is explicitly intended to support extremely small effective batch sizes. LU factorization of $W$ makes determinant evaluation cheap. Zero initialization in the coupling networks makes each layer initially close to the identity map. Gradient checkpointing is used for stability and memory efficiency at high resolution. No additional regularization, including weight decay or dropout, is needed.

For sample quality, Glow uses a lower-bit representation, specifically $5$-bit, together with sampling at reduced temperature $T<1$, written as $p_T(z)\propto p(z)^{T^2}$. Sampling at reduced temperature, for example $T\approx0.7$, is reported to provide the best balance between diversity and sharpness.

## 5. Quantitative performance

Glow is reported to set a new state of the art among non-autoregressive flows on several natural-image benchmarks trained with $8$-bit data [1807.03039].

| Dataset | Glow bpd | Comparison |
|---|---:|---:|
| CIFAR-10 | 3.35 | RealNVP 3.49 |
| ImageNet 32×32 | 4.09 | 4.28 |
| ImageNet 64×64 | 3.81 | 3.98 |
| LSUN Bedroom 64×64 | 2.38 | 2.72 |
| LSUN Tower 64×64 | 2.46 | 2.81 |
| LSUN Church 64×64 | 2.67 | 3.08 |

With $5$-bit dequantization, Glow further achieves $1.67$ bpd on CIFAR-10, $1.99$ bpd on ImageNet $32\times32$, $1.76$ on ImageNet $64\times64$, and $1.03$ on CelebA-HQ $256\times256$ [1807.03039].

The paper also emphasizes throughput and resolution. Glow produces high-fidelity samples at up to $256\times256$ resolution in real time, reported as $0.13$ s/image on a Titan 1080 Ti. A plausible implication is that the model’s architectural tractability was not merely theoretical; it translated into practical synthesis at resolutions that were significant for flow models at the time.

## 6. Latent-space geometry, image manipulation, and name reuse

Glow is presented not only as a likelihood model but also as a generative model with a smooth, semantically meaningful latent space [1807.03039]. Linear interpolation between latent codes of real images produces gradual face morphing without collapse. Post-hoc editing is performed by traversing learned “attribute vectors,” computed as the difference between the mean latent code for positives and negatives of a label. This enables manipulation of smiles, hair color, gender, and age without retraining the generative flow.

The same invertible backbone also motivated later conditional extensions. "Full-Glow: Fully conditional Glow for more realistic image generation" [2012.05846] uses a paired source-and-target Glow architecture in which conditioning information modulates actnorm, invertible $1\times1$ convolution, and affine coupling at every step, indicating that Glow’s invertible design could be repurposed for label-to-image generation.

The name itself is polysemous in the arXiv literature. Distinct works use GLOW or GLoW for unrelated topics, including "Glow: Graph Lowering Compiler Techniques for Neural Networks" [1805.00907], "GLOW : Global Weighted Self-Attention Network for Web Search" [2007.05186], "Gamma Ray Glow Observations at 20-km Altitude" [2107.03181], "GLoW: novel methods for wave-optics phenomena in gravitational lensing" [2409.04606], "GLOW: A Unified Particle Flow Transformer" [2508.20092], and "GLOW: Graph-Language Co-Reasoning for Agentic Workflow Performance Prediction" [2512.15751]. In machine learning, however, the unqualified term “Glow” most often refers to the invertible generative flow model of Kingma and Dhariwal [1807.03039].

## 7. Position within flow-based generative modeling

Glow occupies a specific position within flow-based generative modeling: it preserves exact likelihood and exact invertibility while introducing a learned channel-mixing operator, a stable normalization scheme for deep flows, zero-initialized affine couplings, and a multi-scale factorization strategy [1807.03039]. The model’s reported behavior links these design choices to both quantitative and qualitative outcomes: better log-likelihoods than prior non-autoregressive flows, efficient high-resolution synthesis, and latent manipulations that remain semantically coherent.

Taken together, Glow’s contribution is less a single isolated mechanism than a coordinated recipe: ActNorm, invertible $1\times1$ convolutions, zero-initialized affine couplings, multi-scale splitting, LU parameterization of $W$, and reduced-batch normalization. Within the evidence reported for the model, this combination yields a fast, expressive, and fully invertible generative model that advances both density estimation and controllable image synthesis [1807.03039].

Source: https://www.emergentmind.com/topics/glow-b89d0b18-8cb8-408c-a2ec-489f2a9e65fa