Normalizing Flows Explained

Updated 16 July 2025

Normalizing flows are a class of generative models that use invertible, differentiable transformations to convert simple base distributions into complex target densities.
They leverage the change-of-variables formula with tractable Jacobian determinants to enable efficient sampling and exact likelihood evaluation.
They are applied in diverse fields such as image synthesis, variational inference, reinforcement learning, and probabilistic modeling for practical and scalable inference.

Normalizing flows are a family of generative models that construct complex, high-dimensional probability densities as the result of applying a sequence of invertible, differentiable (diffeomorphic) transformations to a simple base distribution, typically a standard normal or a uniform distribution. The fundamental principle underlying normalizing flows is the change-of-variables formula, which allows for both efficient sampling and exact density evaluation given that the forward and inverse mappings—as well as their Jacobians—are tractable. This approach has enabled normalizing flows to serve as universal density approximators, bridging the gap between explicit likelihood-based generative modeling, sampling, and flexible probabilistic inference.

1. Mathematical Foundations and Model Construction

At the core of normalizing flows is the use of a sequence of invertible maps to transform a sample $z$ from a tractable base distribution $p_Z(z)$ into the more complex target variable $x$ , according to $x = g(z)$ , with $g$ invertible. The corresponding probability density of $x$ is calculated via the change-of-variables formula:

$p_X(x) = p_Z(f(x)) \cdot \left|\det \frac{\partial f(x)}{\partial x}\right|$

where $f = g^{-1}$ denotes the inverse transformation, and the determinant of the Jacobian accounts for local changes in volume. In practice, $g$ is implemented as a composition of $N$ invertible, parameterized transformations $g_i$ :

$g = g_N \circ g_{N-1} \circ \dots \circ g_1$

By the chain rule, the total log-determinant of the Jacobian decomposes as a sum over the sub-transformations:

$\log \det \frac{\partial f}{\partial x} = \sum_{i=1}^N \log \det \frac{\partial f_i(z_i)}{\partial z_i}$

(1908.09257)

For efficient sampling and density evaluation, both the forward and inverse maps, along with the computation of the Jacobian determinant, must be tractable.

2. Architectural Families and Expressivity

Normalizing flows achieve their flexibility by employing a variety of invertible transformations, each trading off expressivity and computational efficiency. The principal architectural categories include:

Elementwise/Linear Flows: Apply scalars or structured linear maps to input dimensions; simple yet limited in capturing dependencies.
Planar and Radial Flows: Modify densities locally or radially, with tractable Jacobians but limited invertibility and expressivity.
Coupling Flows: Partition input and use an affine or nonlinear map on a subset, conditioned on the others (e.g., RealNVP, Glow). Triangular Jacobians enable efficient determinant calculation (1908.09257).
Autoregressive Flows: Each dimension is transformed conditional on preceding ones, ensuring tractable log-determinants (Masked Autoregressive Flows, Inverse Autoregressive Flows). There is a sampling vs. likelihood evaluation trade-off between variants (1908.09257).
Spline-based and Neural Autoregressive Flows: Employ monotonic neural architectures or piecewise rational-quadratic splines for highly expressive invertible maps.
Residual and Continuous Flows: Use residual connections (interpreted as discrete-time ODE steps) or continuous ODE models (Neural ODEs/FFJORD), with the log-density evolution governed by $\frac{d}{dt} \log p(x(t)) = -\operatorname{Tr}\left(\frac{\partial F}{\partial x}\right)$ (1908.09257).
Flows on Manifolds: Geometrically and topologically generalize flows to tori, spheres, general smooth manifolds, and Lie groups by employing recursive, autoregressive, or vector field–parameterized approaches (2002.02428, 2104.14959).

The choice of architecture shapes the flow’s ability to model complex multimodal distributions, accommodate high dimensions, or observe computational constraints.

3. Applications in Probabilistic Modeling and Inference

Normalizing flows have achieved broad applicability across unsupervised, supervised, and scientific domains due to their tractable likelihood, invertibility, and sampling properties:

Density Estimation and Generative Modeling: Flows are trained via maximum likelihood to directly estimate data densities, enabling sample generation that matches intricate target distributions, as demonstrated in image and text generation, audio synthesis, and density modeling for cosmological fields (1912.02762, 2105.12024).
Variational Inference: Flows refine variational approximations by providing flexible posteriors or priors in, for example, variational autoencoders; flows can be embedded in the latent distribution or in the data likelihood (1810.03256, 1912.02762).
Probabilistic Inverse Problems: Flows enable reparameterizable sampling and representation of complex, non-Gaussian posteriors that arise in inverse modeling or likelihood-free inference.
Reinforcement Learning: Flows serve as stochastic policies, Q-function estimators, and occupancy measure models. Their plug-and-play compatibility with maximum likelihood or variational objectives enables integration into imitation learning, offline RL, and goal-conditioned RL while maintaining computational efficiency superior to diffusions or transformers (2505.23527).
Physical and Geometric Sampling: Flows with smooth or manifold-respecting transformations support the sampling of molecular conformations, the acceleration of Markov Chain Monte Carlo (MCMC), and the mitigation of sign problems in lattice field theory (2110.00351, 2101.05755, 2002.02428).

4. Limitations, Relaxations, and Topological Considerations

While normalizing flows are universal approximators of densities under broad conditions, they are subject to structural limitations:

Dimensionality and Topology: Strict invertibility requires input, output, and latent spaces to have matching dimensions and topological type. When the target density’s topology (connected components, boundaries) is incompatible with the base (e.g., a unimodal Gaussian), the flow may exhibit artifacts such as “bridging” between separated modes or “smearing” of density (2309.04433, 2305.02930).
Expressivity of Affine Families: Flows using only affine maps (e.g., standard RealNVP/MAF) see a dramatic increase in expressivity as depth reaches three or more layers, but cannot represent all target densities regardless of depth, as certain components remain fundamentally restricted to affine (hence, Gaussian) forms (2006.00866).
Training and Representational Stability: Normalizing flow objectives can be unstable if the mapped data’s effective dimensionality is lower than nominal. Regularization of the Jacobian (e.g., Tikhonov) prevents degeneracies and ensures robust component estimation (1907.06496).
Likelihood Tractability in Relaxed Flows: To overcome expressivity/topological bottlenecks, recent models combine flows with surjective, stochastic, or diffusion-based layers (e.g., SurVAE, Stochastic Normalizing Flows, Diffusion Normalizing Flows). These relax the need for bijectivity, potentially sacrificing exact density evaluation or sampling efficiency in exchange for greater modeling capacity (2309.04433).

5. Algorithmic and Implementation Advances

Implementation of normalizing flows requires tractable composition of invertible layers, efficient Jacobian determinant computations, and—especially in high-dimensional and manifold cases—storage and computationally efficient architectures.

Efficient Training and Sampling: Flows such as RealNVP and Glow use affine coupling and linear invertible 1×1 convolutions for scalable density modeling and efficient forward/inverse passes. The log-determinant is computed via triangular structures or linear decompositions (PLU/QR) (1908.09257, 2302.12014).
Continuous and Manifold Flows: Neural ODE–based flows generalize invertible maps to the integration of parameterized vector fields; special estimators (Hutchinson’s) compute the requisite divergences for likelihood calculation efficiently, even on non-Euclidean spaces (2104.14959).
Densely Connected/Cross-Unit Flows: Augmenting flows by incrementally padding latent representations with noise and cross-unit coupling increases width and model capacity while retaining invertibility and computational tractability, enabling state-of-the-art density estimation under moderate resource budgets (2106.04627).
Distillation and Flowification: Conditional flows can be distilled into non-invertible—but inference-efficient—models, and standard feed-forward neural network layers (including linear, convolutional, residual) can be adapted into generalized flows by retrofitting inverse passes and likelihood accounting (2106.12699, 2205.15209).
Piecewise/Clustered Flows: Partitioning multi-modal targets into clusters and training separate flows for each enhances representation of complex global structures and avoids the introduction of spurious connections between modes (2305.02930).
Free-Form Flows: Recent training procedures with efficient Jacobian gradient estimators enable maximum likelihood training for arbitrary, dimension-preserving neural networks—removing the architectural constraint of analytical invertibility and allowing for flexible choice of inductive biases (e.g., E(n)-equivariant networks for molecular data), while achieving competitive or superior density modeling (2310.16624).

A table summarizing key architectural classes and their properties:

Architecture	Invertibility Requirement	Expressivity/Trade-offs
Affine Coupling (RealNVP)	Analytic, triangular	Moderate, efficient
Autoregressive (MAF/IAF)	Analytic, sequential	High, trade-off: eval/sampling
Residual/Neural ODE	Contractive/ODE solver	Very high, costlier eval
Manifold/Geometric Flows	Diffeomorphisms on M	High, supports topology
Free-Form Flows	Approx. invertibility	Highest, flexible

6. Recent Developments and Practical Implementations

Several open-source frameworks, such as normflows in PyTorch, provide modular implementations of major normalizing flow architectures—Real NVP, Glow, MAF, Neural Spline Flows, and Residual Flows—facilitating their integration into broader deep learning workflows (2302.12014). Empirical studies across domains, including cosmology, high energy physics, molecular modeling, and reinforcement learning, have validated their adaptability, with flows being particularly well suited for:

Modeling densities and posteriors in high-dimensional spaces with analytic tractability
Flexible policy and value function parameterization in RL, where likelihood computation and differentiable sampling are critical (2505.23527)
Geometric and field-based data, leveraging manifold-aware or smooth flows for stable physical simulation and force computation (2110.00351, 2002.02428)
Density estimation and generative sampling in settings where base-target topology mismatch or data dependencies (non-i.i.d. data) challenge classical methods, handled via piecewise modeling or dependency-aware training objectives (2305.02930, 2209.14933)

7. Outlook and Future Directions

Research on normalizing flows is increasingly focused on relaxing structural constraints to extend expressivity—through stochastic, surjective, or diffusion-inspired model classes; improving computational scalability to high dimensions; developing flows for manifold and structured data (e.g., Lie groups and discrete domains); and further integrating flows with other generative modeling paradigms, such as VAEs and score-based diffusion models (2309.04433). Promising directions include new invertible architectures that are both expressive and tractable, principled regularization using theoretical insights from geometry, and application of flows to novel domains—particularly where exact density computation, efficient sampling, and flexible feature representation are essential.