Expressivity of Bi-Lipschitz Normalizing Flows: A Score-Based Diffusion Perspective

Abstract: Many normalizing flow architectures impose regularity constraints, yet their distributional approximation properties are not fully characterized. We study the expressivity of bi-Lipschitz normalizing flows through the lens of score-based diffusion models. For the probability flow ODE of a variance-preserving diffusion, Lipschitz regularity of the score induces a flow of bi-Lipschitz diffeomorphic transport maps. This ODE bridge allows us to analyze the distributional approximation power of bi-Lipschitz normalizing flows and, conversely, derive deterministic convergence guarantees for diffusion-based transport. Our key idea is to use the probability flow ODE to link regularity of the score to regularity of the induced transport maps. We verify score regularity for broad target densities, including compactly supported densities, Gaussian convolutions of compactly supported measures and finite Gaussian mixtures. We obtain a universal distributional approximation result: Gaussian pullbacks induced by bi-Lipschitz variance-preserving transport maps are $L^1$-dense among all probability densities. For Gaussian convolution targets, we further obtain convergence in Kullback-Leibler divergence without early stopping.

• The paper demonstrates that bi-Lipschitz normalizing flows, induced by VP diffusion ODEs, can universally approximate target densities in L1 without strict uniform Lipschitz limits.
• It establishes explicit regularity criteria linking score-based diffusion models to the expressivity of normalizing flows, ensuring convergence in both L1 and KL for various density classes.
• Empirical experiments reveal that relaxing fixed Lipschitz bounds enhances flow expressivity, offering insights into balancing stability with universal approximation capabilities.

Expressivity of Bi-Lipschitz Normalizing Flows via Score-Based Diffusion

---

Overview

This paper—"Expressivity of Bi-Lipschitz Normalizing Flows: A Score-Based Diffusion Perspective" (2605.06172)—establishes rigorous connections between the regularity properties of transport maps induced by score-based diffusion models and the expressivity of bi-Lipschitz normalizing flows (NFs). By leveraging the probability flow ODE associated with variance-preserving (VP) diffusions, the authors precisely characterize when and how bi-Lipschitz NFs can serve as universal approximators for probability densities. The analysis reveals that, contrary to common beliefs, bi-Lipschitz regularity per se is not an intrinsic expressive limitation—expressivity issues only arise when enforcing a uniform bound on the Lipschitz constants.

---

Bi-Lipschitz Flows, Score-Based Diffusions, and Probability Flow ODEs

Normalizing flows represent complex data distributions as invertible transformations of a latent noise source (e.g., standard Gaussian) via a diffeomorphism $\varphi$. In practice, architectures such as invertible residual networks (iResNets) enforce bi-Lipschitz regularity—both $\varphi$ and its inverse have bounded Lipschitz constants—to guarantee stability and invertibility.

However, stringent uniform bi-Lipschitz bounds can severely constrain the complexity of the learned map, especially when pushing-forward measures with disconnected or highly heterogeneous support.

The key tool in this work is the deterministic probability flow ODE associated with score-based VP diffusions. For a diffusion process with score function $s_t(x) = \nabla_x \log p_t(x)$, the corresponding ODE takes the form:

$$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$

where the time-dependent vector field is directly controlled by the regularity of $s_t$. Under mild conditions on the target, the induced end-to-end transport map from $p_H$ to $p_Z$ (Gaussian) is a bi-Lipschitz diffeomorphism, though the associated Lipschitz constants can depend on the time window of the diffusion.

---

Figure 1: Example of a compactly supported pdf $p_H$ and monotone transport $f$ to the standard Gaussian $p_Z$, satisfying $\varphi$0. The exact transport is not a global diffeomorphism on $\varphi$1, and neither $\varphi$2 nor $\varphi$3 is globally Lipschitz. Discontinuities can only be approximated by bi-Lipschitz maps. A bi-Lipschitz map cannot exactly transform the connected full support of the Gaussian into the disconnected support of $\varphi$4. Colored curves show iResNets trained with increasing Lipschitz bounds.

---

Distributional Expressivity—Main Theoretical Results

The core contribution is a suite of theorems rigorously linking score regularity to the expressivity of bi-Lipschitz normalizing flows from a distributional (rather than function-approximation) perspective. The strong results are as follows:

• Universal $\varphi$5-approximation: For any target density $\varphi$6, there exists (for any accuracy $\varphi$7) a bi-Lipschitz $\varphi$8 diffeomorphism $\varphi$9—induced as the end-point map of a VP flow—such that pushing the standard Gaussian through $s_t(x) = \nabla_x \log p_t(x)$0 yields a density within $s_t(x) = \nabla_x \log p_t(x)$1 of $s_t(x) = \nabla_x \log p_t(x)$2 in $s_t(x) = \nabla_x \log p_t(x)$3. This is achieved without enforcing uniform (i.e., architecture-level) bounds on the Lipschitz constants [Theorem 1, Corollary: Universal Approximation].
• KL convergence with smooth targets: If $s_t(x) = \nabla_x \log p_t(x)$4 is a Gaussian convolution of compact support or a finite Gaussian mixture, the associated VP flow satisfies uniform-in-time and -space Lipschitz score bounds, and the pullback density converges in KL as well as $s_t(x) = \nabla_x \log p_t(x)$5 [Corollary: Score Regularity for Gaussian Mixtures].
• The expressivity limitations for flows with uniform Lipschitz constraints (typically realized in actual architectures like iResNets with fixed spectral norm budgets) are shown to be strictly quantitative. As the constraint $s_t(x) = \nabla_x \log p_t(x)$6, approximation power increases, but only in the limit does the family become dense in $s_t(x) = \nabla_x \log p_t(x)$7.
• The analysis produces concrete regularity criteria for the target: compactly supported, log-concave, Gaussian-convoluted densities, and mixtures thereof all admit the desired controlled score regularity for the corresponding diffusion.

---

Numerical Illustrations

Empirical experiments substantiate and visualize the theoretical findings:

• 1D and 2D Density Learning: The authors train iResNets with varying Lipschitz constants ($s_t(x) = \nabla_x \log p_t(x)$8) and VP-based score models on diverse targets: compact supports, disconnected mixtures, and high-regularity densities. As $s_t(x) = \nabla_x \log p_t(x)$9 increases ($$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$0), iResNets more closely capture the true transports and densities, especially for complicated/disconnected supports.

Figure 2: Inverse transports learned by iResNets compared to the ground truth inverse for different levels of Lipschitz constraint $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$1 and for SDM. Accuracy degrades substantially for low $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$2 in the presence of discontinuities.

Figure 3: Ground truth target $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$3 and corresponding learned densities $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$4 from iResNets $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$5 at differing $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$6. SDMs robustly capture sharp/complex supports even without explicit architectural constraints.

• Score Regularity Dynamics: The empirical estimates $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$7 are plotted as a function of time during the VP process. For compactly supported or log-concave $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$8, $$\frac{dx}{dt} = -\frac{1}{2} x - \frac{1}{2} s_t(x)$$9 is finite for $s_t$0 but can diverge as $s_t$1, in line with the theoretical analysis.

Figure 4: Approximation errors $s_t$2 and KL divergence across $s_t$3 for iResNets and for SDMs. As $s_t$4 increases, expressivity saturates. SDM exhibits superior performance for disconnected/complex supports.

---

Implications and Theoretical Significance

This work provides a definitive answer to a standing open question: bi-Lipschitz constraints are not fundamental obstructions to density expressivity—the observed limitations in practice stem from architectural choices imposing uniform (global) limits on the Lipschitz constant. By contrast, the theoretically expressive class comprises all bi-Lipschitz diffeomorphisms with possibly unbounded constants in the approximation limit.

The explicit connection to score-based diffusions yields critical insights into the metric relationships between the score field, transport regularity, and pushforward expressivity. Further, the paper positions VP-based flows as a concrete family achieving universality in $s_t$5 for deterministic map-based generative models.

Practically, the results indicate the need for adaptive or locally-varying regularity during training if one wishes to reconcile invertibility, stability, and strong universal approximation. They also clarify the obstacles for current invertible neural net architectures: fixed, global regularity constraints (e.g., low spectral norm) can never represent all targets to arbitrary precision.

---

Extensions and Open Problems

While the analysis is at the level of function spaces (not architectures), the established hierarchy between the strength of score regularity and corresponding convergence metrics (KL, $s_t$6) sets a clear target for architecture-aware results. Understanding how well architectural approximators (e.g., coupling flows, iResNets) can distill VP flows, or how their expressivity evolves as one relaxes explicit global regularity constraints, is a concrete direction for future work.

Moreover, the results raise the prospect of blending deterministic and stochastic flows, or adapting the regularity along the diffusion time axis, as a means to achieve both stability and universal approximation in practice.

---

Conclusion

By leveraging the probability flow ODE of variance-preserving diffusion models, this work precisely establishes the $s_t$7-density (and in many cases KL-density) of the bi-Lipschitz normalizing flow family. The critical theoretical advance is the explicit, verifiable connection between score regularity and distributional approximation power, together with constructive density results for natural and expressive target classes. Empirically, the work clarifies the interplay between explicit regularity constraints and expressivity in modern generative modeling. These findings guide both the practical design of generative models and the theoretical analysis of their capabilities and limitations.