Diffusion Normalizing Flow Overview

• Diffusion normalizing flow is a generative modeling paradigm that combines invertible flow structures with stochastic diffusion processes via SDEs to capture complex distributions.
• It employs joint training objectives like maximum likelihood and score matching to ensure robust density estimation and efficient sampling across diverse data.
• Hybrid architectures integrating deterministic and stochastic elements enhance expressivity and speed, advancing applications in image generation, PDE solving, and Bayesian inference.

A diffusion normalizing flow is a generative modeling paradigm that integrates the expressive, invertible mapping structure of normalizing flows with the stochastic regularization and likelihood theory of diffusion processes, typically formulated via stochastic differential equations (SDEs). These models combine the statistical efficiency and invertibility of flows with the flexibility and mode-covering capacity of diffusion, enabling the modeling of highly complex, high-dimensional distributions with both sharp and smooth features. Diffusion normalizing flows generalize deterministic normalizing flows to stochastic, measure-preserving Markov processes, admit both maximum likelihood and score-matching training objectives, and support efficient sampling and likelihood estimation regimes that interpolate between classical flows and diffusion models. Recent research has extended these ideas to improved pathwise training, efficient surrogates for Fokker–Planck equations, joint diffusion–flow architectures, and Markov-chain-based frameworks.

1. Mathematical Foundations and Core Formulations

The core object in diffusion normalizing flows is a pair of Markov processes (or SDEs), typically specified as a forward (noising) and a corresponding reverse (denoising) process:

• Forward SDE:

$$dX_t = f_\theta(X_t, t)\,dt + g(t)\,dW_t, \qquad X_0 \sim p_{\text{data}}$$

where $f_\theta$ is a learnable drift, $g(t)$ a noise schedule, and $W_t$ a standard Brownian motion.

• Reverse SDE:

$$dX_t = [f_\phi(X_t, t) - g^2(t)\,\nabla_x \log p_t(X_t)]\,dt + g(t)\,d\bar W_t, \qquad X_T \sim \mathcal N(0, I)$$

Here, $p_t$ is the marginal density at time $t$ in the forward process, and the score function $\nabla_x \log p_t(x)$ is often approximated via a neural network.

• In discrete implementations, the process is often realized as a sequence of invertible or stochastic mappings—Markov kernels or SDE steps—that together define a path measure transforming base noise distributions into complex data distributions (Hagemann et al., 2021, Zhang et al., 2021).
• The normalization constraint on probability densities, especially for time-dependent Fokker–Planck PDEs,

$$\frac{\partial p}{\partial t} = -\nabla_x \cdot [p(x, t)\,\mu(x, t)] + \nabla_x \cdot [\nabla_x \cdot (p(x, t)D(x,t))]$$

is naturally enforced in these frameworks through the change-of-variables formula and measure-preserving properties of the flow (Liu et al., 2023, Saporiti et al., 19 Mar 2026).

• Many recent models augment the deterministic invertible structure of normalizing flows with stochastic diffusion steps, yielding the “stochastic normalizing flow” (SNF) formalism, where layers may be both invertible and diffusive, and the model class sits in a hierarchy interpolating between flows, diffusion models, and VAEs (Hagemann et al., 2021).

2. Training Objectives, Score Matching, and Maximum Likelihood

Diffusion normalizing flows support multiple complementary training principles:

• Maximum Likelihood and Path-Measure KL:

Joint training of the forward and reverse SDEs seeks to minimize the Kullback–Leibler divergence between the path measures they induce,

$\mathrm{KL}[p_F(\tau)\,\Vert\,p_B(\tau)]$

ensuring that the reverse process generates samples matching the target data measure, with the loss often reduced to evaluating sequence-wise noise contributions and terminal likelihoods (Zhang et al., 2021).

• Score Matching and Denoising Objectives:

Score-based approaches optimize the denoising score matching objective,

$f_\theta$0

where $f_\theta$1 is a score network and $f_\theta$2 is the marginal density at noise level $f_\theta$3 (Gong et al., 2021, Zand et al., 2023). For diffusion score matching (DSM), a state-dependent diffusion matrix $f_\theta$4 is sometimes incorporated, allowing geometric flexibility (Gong et al., 2021).

• Joint or Self-Supervised Losses:

In physical or high-dimensional PDE contexts, “physics-informed” flows (PINF) define a self-supervised MSE between direct neural predictions and ODE-integrated density evaluations along characteristics, ensuring robust normalization and positivity (Liu et al., 2023).

• Proximal Wasserstein Flows (JKO-iFlow):

Some models implement the Jordan–Kinderlehrer–Otto (JKO) proximal discretization of the Wasserstein gradient flow for KL divergence, leading to block-wise residual flow architectures approximating diffusion trajectories deterministically (Xu et al., 2022).

3. Model Architectures: Flow, Diffusion, and Hybrid Design

A wide range of architectures has been proposed:

• Neural SDE Flows and CNF Hybrids:

Both forward and reverse SDEs may be parameterized by neural networks, extending continuous normalizing flows (CNFs) by including diffusion terms. In the limit $f_\theta$5, one recovers deterministic CNFs as a special case (Zhang et al., 2021, Liu et al., 2023).

• Markov-Chain Hybrid Layers:

Generalized models compose deterministic invertible layers, SDE/diffusion kernels, and even Markovian sampling steps (e.g., Langevin, Metropolis-Hastings), each with explicit or implicit invertibility, and train via a unified pathwise KL loss (Hagemann et al., 2021).

• Physics-Informed Flows (PINF):

For Fokker–Planck-type problems, neural networks approximate the log-density and its gradients, enabling incorporation of both drift and diffusion terms in their learned vector fields, yielding mesh-free, causality-free, and high-dimensional solvers (Liu et al., 2023).

• Neural Galerkin Flows:

These use structure-preserving normalizing flows as parametric surrogates for transition probability densities, evolving via ODEs derived from Galerkin variational principles and leveraging adaptive residual minimization (Saporiti et al., 19 Mar 2026).

• Deterministic–Stochastic Hybrids:

Methods such as DiNof and INDM combine stochastic diffusion and deterministic normalizing flows, often alternating or serializing the two to maximize expressivity and sampling speed (Zand et al., 2023, Kim et al., 2022).

4. Sampling, Likelihood Evaluation, and Computational Practices

Sampling and likelihood computation are central to diffusion normalizing flows:

• Sampling Procedures:

Often proceeds by generating noise from a base distribution (e.g., $f_\theta$6), transforming via a (possibly invertible) flow, and then simulating or solving the (reverse) SDE to denoise or map to the data distribution (Zhang et al., 2021, Zand et al., 2023).

• Efficiency and Step-Size Tradeoffs:

Stochasticity relaxes bijectivity but increases expressive power; with moderate numbers of discretization steps (10–100), diffusion normalizing flows achieve efficient sampling while preserving data fidelity, substantially outperforming fixed-step DDPMs in speed/quality tradeoffs (Zhang et al., 2021).

• Likelihoods:

Exact likelihood and change-of-variables evaluations remain available in the deterministic and ODE solvers, while variational bounds apply for stochastic layers. The probability flow ODE yields pathwise normalizing flows with tractable log-density computations (Zhang et al., 2023).

5. Expressivity, Theoretical Properties, and Comparative Assessment

Diffusion normalizing flows provide notable theoretical and empirical advantages:

• Expressivity:

By mixing deterministic and stochastic layers, these models can represent distributions beyond those accessible to deterministic flows or pure diffusion models. For instance, composition increases the class of attainable generative distributions, permitting sharper boundaries and disconnected support (Zhang et al., 2021, Zand et al., 2023, Kim et al., 2022).

• Theoretical Guarantees:

Many frameworks provide rigorous asymptotic convergence guarantees (e.g., exponential decay in KL divergence under log-Sobolev conditions) (Zhang et al., 2023). Joint MLE and score-matching objectives yield strong statistical consistency (Kim et al., 2022).

• Surrogate Density Modeling:

For PDE and SDE surrogate modeling, offline-trained diffusion normalizing flows enable rapid online evaluation for many-query tasks, maintaining mass conservation and positivity (Saporiti et al., 19 Mar 2026, Liu et al., 2023).

• Performance:

On density estimation (tabular, 2D synthetic) and image generation tasks (MNIST, CIFAR-10, CelebA), diffusion normalizing flows match or outperform leading flow and diffusion methods in both FID and NLL metrics (e.g., FID=2.01 on CIFAR-10, FID=1.75 on CelebA-64) and offer robustness to discretization (Zhang et al., 2021, Kim et al., 2022, Zand et al., 2023).

6. Generalizations and Unifying Frameworks

Recent research situates diffusion normalizing flow models within unified formalisms:

• Markov Chain Frameworks:

All principal generative architectures—deterministic flows, stochastic flows (diffusion flows), VAEs, and MCMC—can be seen as layered Markov chains with forward/reverse kernels, Radon–Nikodym derivatives, and unified training objectives, allowing hybrid composition and fine-grained control over expressivity and sampling (Hagemann et al., 2021).

• Geometry and Riemannian Extensions:

The relation between diffusion score matching and ordinary score matching is formalized via Riemannian metric transformations induced by flows, revealing geometric structure and optimization properties (Gong et al., 2021).

• Hybrid and Amortized Inference:

Joint training of flows and diffusion models for inverse or corrupted-data tasks combines fast amortized inference with score-based diffusion priors, enabling high-fidelity generative modeling in challenging regimes (e.g., corrupted, noisy, or incomplete data) (Wang et al., 2024).

7. Applications, Limitations, and Future Directions

Diffusion normalizing flows have demonstrated success in:

• Adaptive solution of high-dimensional Fokker–Planck equations, serving as mesh-free surrogates for scientific computing (Liu et al., 2023, Saporiti et al., 19 Mar 2026).
• Fast, likelihood-based, and high-quality generative modeling for images and tabular data (Zhang et al., 2021, Kim et al., 2022, Zand et al., 2023).
• Amortized inference and Bayesian inverse problems under data corruption (Wang et al., 2024).

Key limitations include increased model and implementation complexity versus pure CNF or DDPM approaches, substantial offline training requirements for surrogate PDE applications, and, in some cases, challenges in scaling to extremely high-dimensional raw data. Directions for further development include improved adaptivity, scalable architectures, theoretical error/robustness analysis, and large-scale benchmarks for hybrid models.

---

References: (Zhang et al., 2021, Hagemann et al., 2021, Liu et al., 2023, Zand et al., 2023, Kim et al., 2022, Xu et al., 2022, Zhang et al., 2023, Saporiti et al., 19 Mar 2026, Gong et al., 2021, Wang et al., 2024, Chen et al., 21 Apr 2026).