Continuous-Time Normalizing Flows (CNF)

Continuous-Time Normalizing Flows (CNF) are a family of probabilistic generative models that define flexible, invertible mappings between probability distributions using the machinery of ordinary differential equations (ODEs) parameterized by neural networks. CNFs have become central to modern approaches across generative modeling, simulation-based inference, and stochastic process learning, owing to their expressivity, tractable density estimation, and applicability to a wide variety of data modalities, including those with irregular time structure or non-Euclidean geometry.

1. Mathematical Foundations and Model Architecture

A Continuous-Time Normalizing Flow defines a deterministic dynamical system for a random variable $z(t) \in \mathbb{R}^d$: $$\frac{d z(t)}{dt} = f_\theta(z(t), t), \quad z(0) \sim p_0(z)$$ where $f_\theta$ is a neural network parameterizing the dynamics and $p_0$ is a base distribution (typically Gaussian).

Under this flow, the probability density at time $t$ evolves according to the instantaneous change-of-variables formula: $$\frac{d \log p(z(t))}{dt} = - \operatorname{Tr}\left( \frac{\partial f_\theta(z(t), t)}{\partial z} \right)$$ Integrating this ODE from $t_0$ to $t_1$ yields the transformation: $$\log p(z(t_1)) = \log p(z(t_0)) - \int_{t_0}^{t_1} \operatorname{Tr}\left( \frac{\partial f_\theta(z(t), t)}{\partial z} \right) dt$$ This enables both forward sampling (by integrating from base to target) and exact computation of log-densities (by tracking the trace term through integration).

Sampling and density evaluation rely on numerical ODE solvers (e.g., Runge-Kutta), with trace computation typically handled by stochastic estimators such as Hutchinson's trick for scalability in high dimensions.

2. Expressivity, Universality, and Theoretical Properties

Continuous-time parameterization endows CNFs with universal approximation properties for diffeomorphic transformations: any smooth, invertible mapping between distributions can, in principle, be learned by an appropriately expressive neural ODE (Papamakarios et al., 2019 ). This flexibility allows modeling of multimodal, highly correlative, and non-factorial data distributions.

Recent theoretical work rigorously analyzes the statistical and numerical errors of CNFs, especially in learning target distributions from finite samples. Under mild regularity assumptions (bounded support, strong log-concavity, or mixture-of-Gaussians targets), CNFs trained by flow matching achieve provable non-asymptotic bounds in Wasserstein-2 distance, consolidating the reliability of CNFs in practical settings (Gao et al., 31 Mar 2024 ). Regularity of the learned velocity fields, the treatment of discretization and early stopping errors, and uniform approximation properties via deep ReLU networks are all essential to convergence guarantees.

$$\mathbb{E}\left[ \mathcal{W}_2(\hat{p}_{1-\underline{t}}, p_1) \right] = \widetilde{\mathcal{O}}\left(n^{-1/(d+5)}\right)$$

where $\hat{p}_{1-\underline{t}}$ is the CNF estimator, $p_1$ is the target, and $n$ is the number of samples.

3. Training Objectives and Computational Strategies

Likelihood-Based Training

The archetypal objective maximizes the exact log likelihood: $$\log p(x) = \log p_0(z(0)) - \int_{0}^T \operatorname{Tr}\left(\frac{\partial f_\theta(z(t), t)}{\partial z}\right) dt$$ For high-dimensional flows, parameterizations of $f_\theta$ and efficient trace estimation are crucial. Designs leveraging neural potentials (Onken et al., 2020 ) and architectural constraints (e.g., invertible coupling, 1x1 convolutions) are commonly used.

Flow Matching

Flow matching provides a regression framework for learning the velocity field $v^*$ that guides the ODE, sidestepping the need for the intractable computation of likelihoods in some contexts. Given a path between source $z(0)$ and target $x(1)$, often from linear interpolation, flow matching fits $f_\theta$ to minimize

$$\mathbb{E}_{t, z(t)} \left[ \| f_\theta(z(t), t) - v^*(z(t), t) \|_2^2 \right]$$

with theoretical guarantees on approximation and generalization (Gao et al., 31 Mar 2024 ). This approach underpins highly scalable diffusion and flow-based generative models in large-scale applications.

Regularization and Numerical Stability

The number of function evaluations (NFEs) during ODE integration presents computational bottlenecks for CNFs. Methods such as Trajectory Polynomial Regularization (TPR) penalize non-polynomial trajectories, reducing solver effort without harming approximation quality (Huang et al., 2020 ). Optimal transport (OT) theory has also guided the use of kinetic energy and Hamilton–Jacobi–BeLLMan regularizers to control path straightness and tractability (Onken et al., 2020 ). Temporal optimization strategies (e.g., TO-FLOW) dynamically adjust integration time to balance computational and modeling costs (Du et al., 2022 ).

4. Extensions: Conditioning, Manifolds, and Structured Data

Conditional and Structured Flows

Conditional CNFs incorporate side information (e.g., context, class labels) by allowing the velocity field or the base density to depend on input features, resulting in conditional densities $p(y|x)$ with rich inter-dimensional modeling capability. Example applications include conditional image generation, super-resolution, and structured spatio-temporal prediction (Winkler et al., 2019 , Zand et al., 2021 ). Partitioning strategies (supervised/unsupervised latent splits), as in InfoCNF, and innovative factorization across temporal or spatial domains, enable highly efficient and effective modeling (Nguyen et al., 2019 ).

CNFs on Manifolds

For non-Euclidean data, CNFs have been generalized to operate on smooth manifolds, including spheres, Lie groups, and product spaces (Falorsi, 2021 , Ben-Hamu et al., 2022 ). This requires parameterizing vector fields intrinsically using local frames or equivariant neural architectures, and adapting the change-of-variables formula to account for Riemannian divergence. Scalable unbiased estimators for geodesic divergence (e.g., manifold Hutchinson’s estimator) permit efficient density computation.

The Probability Path Divergence (PPD) offers an alternative to likelihood maximization on manifolds, providing scalable divergences for matching probability densities along prescribed paths without repeated ODE solutions (Ben-Hamu et al., 2022 ).

5. Applications and Empirical Results

CNFs have demonstrated strong empirical performance in a range of domains:

• Density Estimation & Generative Modeling: CNFs match or exceed discrete flows and variational inference baselines in density estimation on standard tabular and image datasets (e.g., OT-Flow achieves competitive log-likelihoods with much fewer parameters and up to 24x speedup in inference (Onken et al., 2020 )).
• Irregular Time Series: Dynamic CNF architectures enable exact and efficient modeling of continuous-time stochastic processes, providing native support for irregularly sampled data in healthcare, finance, and physical simulations (Deng et al., 2020 ).
• Molecular & Scientific Simulation: CNFs are capable of learning equilibrium distributions and complex conformational spaces for molecular systems, with shortcut regression distilling deep CNFs into efficient invertible mappings (Rehman et al., 1 Jun 2025 ).
• Adversarial Purification: Recent methods such as FlowPure leverage CNFs for robust purification and detection of adversarial examples, outperforming diffusion-based counterparts in both accuracy and sample fidelity (Collaert et al., 19 May 2025 ).
• Lattice Gauge Theories: Group-equivariant CNFs have been developed for sampling configurations in lattice gauge models, maintaining gauge invariance and proving effective for high-dimensional matrix Lie group spaces (Gerdes et al., 17 Oct 2024 ).

6. Limitations and Future Directions

Although CNFs are expressive and supported by strong theoretical guarantees, they present practical challenges:

• Compute and Memory: ODE integration and trace estimation can be computationally intensive in high dimensions, though regularization, architectural innovations, and specialized solvers continue to reduce costs (Onken et al., 2020 , Huang et al., 2020 ).
• Likelihood and OOD Detection: Like other likelihood-based models, CNFs may assign high probabilities to out-of-distribution samples, limiting direct application to anomaly detection without further correction (Voleti et al., 2021 ).
• Hyperparameter Tuning: Regularization parameters in OT-based CNFs require careful adjustment, though methods leveraging the JKO scheme now allow for robust and tuning-free training (Vidal et al., 2022 ).
• Scalability to Ultra-High Dimensions: Pathwise divergences and manifold CNFs extend scalability, but challenges remain for image-scale data or complex geometric topologies (Ben-Hamu et al., 2022 ).

Promising research directions include hybrid training objectives, further theory connecting CNFs to stochastic flows, incorporating domain-specific symmetries, multi-resolution architectures for images (Voleti et al., 2021 ), and new application domains in structured scientific data and simulation-based inference.

Summary Table: Core Properties and Innovations of CNFs

Continuous-Time Normalizing Flows are a foundational technique underpinning a wide spectrum of current research in generative modeling and simulation-based inference. Their theoretical grounding, practical scalability, and versatility for structured and geometric data continue to drive active development and application within and beyond the machine learning community.