Flow++: Advanced Flow-Based Generative Model

• Flow++ is a flow-based generative model that learns an invertible mapping for efficient, parallel likelihood evaluation and sample generation.
• It employs variational dequantization to transform discrete image data, achieving approximately 0.13 bits/dim improvement on CIFAR10 compared to uniform methods.
• Enhanced with mixture-of-logistics couplings and self-attention conditioning, Flow++ attains state-of-the-art density estimation while significantly improving sample quality and speed.

Flow++ is a state-of-the-art flow-based generative model designed for unconditional density estimation on standard image benchmarks, distinguished by its use of variational dequantization, highly expressive mixture-of-logistics coupling transforms, and conditioning networks incorporating self-attention. Flow++ effectively narrows the historical performance gap between flow-based and autoregressive models in exact likelihood modeling, improving both expressiveness and sample quality while retaining the efficiency of parallel ancestral sampling (Ho et al., 2019).

1. Flow-Based Generative Models: Principles and Limitations

Flow-based generative models define an invertible mapping

$$f : x \;\longleftrightarrow\; z \quad,\quad z\sim\mathcal N(0,I)$$

such that exact data log-density is computable as

$$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$

and generation is enabled by sampling $z\sim\mathcal N(0,I)$ and applying $x=f^{-1}(z)$. The invertibility and tractable Jacobian determinants of each flow layer ensure efficient likelihood evaluation and parallel sampling.

Compared to autoregressive models—which exploit flexible conditionals via $p(x)\!=\!\prod_d p(x_d\mid x_{<d})$ at the cost of inherently sequential $O(D)$ sampling—earlier flows such as NICE, RealNVP, and Glow sample in $O(1)$ parallel steps but have lagged significantly behind autoregressive models (e.g., PixelCNN, PixelRNN) in negative log-likelihood due to limitations in their coupling transformations, dequantization, and conditioning architectures.

2. Variational Dequantization for Discrete Data

Observed image data $x\in\{0,\dots,255\}^D$ are discrete. Modelling these with a continuous density $p_\theta(y)$ necessitates a dequantization procedure, turning $x$ into $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$0, $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$1.

Uniform Dequantization

Standard flow models employ uniform noise:

$$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$2

and maximize a log-likelihood lower bound via Jensen’s inequality: \begin{align} \mathbb{E}{y\sim p{\rm data}}[\log p_\theta(y)] &= \sum_x P_{\rm data}(x)\int_{[0,1)^D}!\log p_\theta(x+u)\,du \nonumber\ &\le \sum_x P_{\rm data}(x)\,\log\int_{[0,1)^D} p_\theta(x+u)\,du = \mathbb{E}{x\sim P{\rm data}}[\log\,P_\theta(x)] \end{align} This approach forces $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$3 to be flat within each unit hypercube.

Variational Dequantization

Flow++ introduces a learned dequantization distribution $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$4 with support on $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$5, treated as a variational posterior:

$$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$6

The combined flow model and dequantizer are jointly optimized:

$$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$7

When $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$8 is chosen as a conditional flow (i.e., $$\log p_\theta(x) \;=\; \log\mathcal N\bigl(f(x);0,I\bigr) \;+\;\sum_{i=1}^L \log\Bigl|\det\,\tfrac{\partial f_i}{\partial f_{i-1}}\Bigr|$$9, $z\sim\mathcal N(0,I)$0), pathwise gradients are available. The uniform dequantization bound is recovered as a special case. This variational approach produces a model less restricted by cell-wise uniformity and increases likelihood performance by approximately 0.13 bits/dim on CIFAR10 (Ho et al., 2019).

3. Coupling Layers: Mixture-of-Logistics Transforms

Traditional affine coupling in flows operates as follows: \begin{align} x = (x_1, x_2)\quad &\mapsto\quad y_1 = x_1,\quad y_2 = x_2\exp(a_\theta(x_1)) + b_\theta(x_1) \end{align} where $z\sim\mathcal N(0,I)$1 and $z\sim\mathcal N(0,I)$2 are outputs of a neural net on $z\sim\mathcal N(0,I)$3. Flow++ replaces this affine transformation with a richer, nonlinear, elementwise transformation based on the cumulative distribution function of a mixture of $z\sim\mathcal N(0,I)$4 logistics: \begin{align} y &= \sigma^{-1}\Bigl(\mathrm{MixLogCDF}(x; \pi, \mu, s) \Bigr)\exp(a) + b \ \mathrm{MixLogCDF}(x; \pi, \mu, s) &= \sum_{i=1}^K \pi_i\, \sigma\Bigl((x - \mu_i)\exp(-s_i)\Bigr) \end{align} Parameters $z\sim\mathcal N(0,I)$5 are computed by a network conditioned on $z\sim\mathcal N(0,I)$6. This coupling first warps $z\sim\mathcal N(0,I)$7 through the mixture-of-logistics CDF, maps to $z\sim\mathcal N(0,I)$8 via the inverse sigmoid, and applies an affine scaling. The monotonicity of the CDF ensures invertibility, with the inverse and Jacobian tractable by leveraging the mixture’s density. This enables each coupling layer to effect more complex marginal transformations than affine functions alone, yielding higher performance (≈0.03 bits/dim improvement over affine forms) (Ho et al., 2019).

4. Conditioning Networks Enhanced with Self-Attention

In earlier flow architectures, the neural network outputting coupling parameters (the "stuffer" network) was entirely convolutional. Flow++ augments this network with multi-head self-attention, markedly increasing expressiveness while maintaining tractability. For each residual block in the conditioning pipeline, there is an alternation between:

• $z\sim\mathcal N(0,I)$9 convolutional gated residual units (input → nonlinearity → Conv$x=f^{-1}(z)$0 → nonlinearity → gated linear unit)
• Multi-head self-attention gated residual units (input → Conv$x=f^{-1}(z)$1 → MultiHeadSelfAttention → gated linear unit)

Layer normalization follows each such block. A stack of 10 such blocks for datasets like CIFAR10 enables flexible parameterization, capturing both local and long-range dependencies essential for effective image transformation across coupling layers. Empirical ablation confirms the self-attention enhancement yields ≈0.03 bits/dim improvement (Ho et al., 2019).

5. Optimization, Sampling, and Implementation

Flow++ optimizes the sum of the variational dequantization lower bound and standard flow log-likelihood over continuous $x=f^{-1}(z)$2, with pathwise backpropagation through both primary flow $x=f^{-1}(z)$3 and dequantization flow $x=f^{-1}(z)$4. Implementation includes:

• ActNorm layers for channel-wise affine normalization with data-dependent initialization
• Invertible $x=f^{-1}(z)$5 convolutions before each coupling layer
• Checkerboard and channel squeeze/unsqueeze operations
• Weight normalization for convolutional kernels
• Adam or RMSProp optimizers
• Importance sampling with up to 16,384 samples at evaluation for a tighter dequantization bound

At generation, standard ancestral flow sampling is performed: sample $x=f^{-1}(z)$6, invert via $x=f^{-1}(z)$7, discard dequantization noise, and round to the nearest discrete grid point.

6. Empirical Evaluation and Model Comparison

On unconditional density estimation (bits per dimension), Flow++ demonstrates the following empirical results:

Model	CIFAR10	ImageNet 32×32	ImageNet 64×64
RealNVP (flow)	3.49	4.28	–
Glow (flow)	3.35	4.09	3.81
IAF-VAE (flow)	3.11	–	–
Flow++	3.08	3.86	3.69
PixelCNN (AR)	3.14	–	–
Gated PixelCNN	3.03	3.83	3.57
PixelCNN++	2.92	–	–
PixelSNAIL (AR)	2.85	3.80	3.52

With a single importance sample at training, Flow++ attains 3.12 bits/dim on CIFAR10, outperforming Glow and RealNVP by 0.27 and 0.41 bits/dim, respectively. This improvement is attributable to the cumulative effects of variational dequantization, mixture-CDF couplings, and self-attention conditioning. Flow++ also generates samples whose perceptual quality approaches autoregressive PixelCNN models while offering an order-of-magnitude faster sampling rate (≈0.04s per image for Flow++ versus ≈0.4s for PixelCNN++) (Ho et al., 2019).

7. Significance and Summary

Flow++ achieves state-of-the-art non-autoregressive generative modeling on standard image datasets by jointly learning a data-dependent dequantization distribution, deploying mixture-of-logistics coupling layers, and enhancing parameterization with self-attention-based conditioning networks. These innovations enable Flow++ to closely approach the density modeling performance of autoregressive models while preserving the computational advantages of efficient, parallelizable flow-based sampling (Ho et al., 2019).