--- title: Bi-Lipschitz Normalizing Flows Explained url: https://www.emergentmind.com/topics/bi-lipschitz-normalizing-flows type: topic --- # Bi-Lipschitz Normalizing Flows Explained Searching arXiv for recent and foundational papers on bi-Lipschitz normalizing flows. arXiv query: bi-Lipschitz normalizing flows expressivity monotone operators B-spline flows Bi-Lipschitz normalizing flows are normalizing flows whose forward map and inverse are both globally Lipschitz with finite bounds. In the standard formulation, an invertible map $F : X \subset \mathbb{R}^d \to Z \subset \mathbb{R}^d$ is $(L_1,L_2)$-bi-Lipschitz if \[ \forall x,y \in X:\ \|F(x)-F(y)\| \le L_1\|x-y\|, \qquad \forall u,v \in Z:\ \|F^{-1}(u)-F^{-1}(v)\| \le L_2\|u-v\|, \] equivalently, \[ \frac{1}{L_2}\|x-y\| \le \|F(x)-F(y)\| \le L_1\|x-y\|. \] With a base density $q$ and differentiable bijection $F$, the induced density is $\hat p(x)=q(F(x))\,|\det D F(x)|$ [2107.07232]. This class is central because many modern flow architectures are either bi-Lipschitz by design or trained under Lipschitz control to ensure numerical stability, tractable inverses, and robust optimization, yet the same regularity imposes nontrivial expressivity constraints [2107.07232]. Subsequent work has refined both sides of this picture: negative results under uniform Lipschitz constraints, positive $L^1$-density results when Lipschitz bounds are allowed to depend on the approximation target, and architectural constructions that preserve explicit bi-Lipschitz control while enlarging the class of realizable transports [2605.06172]. ## 1. Definitions, geometric meaning, and likelihood structure For differentiable bi-Lipschitz maps, the Jacobian is uniformly controlled. If $F$ is differentiable and $L_1$-Lipschitz, then $\|D F(x)\|_{\mathrm{op}} \le L_1$ for all $x$, while $F^{-1}$ being $L_2$-Lipschitz implies $\sigma_{\min}(D F(x)) \ge 1/L_2$. Hence \[ (1/L_2)^d \le |\det D F(x)| \le L_1^d \] for all $x$ [2107.07232]. Equivalent statements appear in broader flow-theoretic treatments: if a map $f:\mathbb{R}^d\to\mathbb{R}^d$ is bi-Lipschitz with constants $m,L$, then singular values of $J_f(x)$ lie in $[m,L]$ almost everywhere and \[ m^d \le |\det J_f(x)| \le L^d \] [2010.01155, 2102.06539, 2304.04555]. These determinant bounds encode bounded local volume distortion. Regions cannot be expanded arbitrarily and cannot be collapsed arbitrarily. In generative modeling terms, the pushforward density cannot exhibit unlimited dynamic range relative to the base distribution because both the latent density term and the Jacobian term are uniformly constrained [2107.07232]. This is the geometric core of why bi-Lipschitzness stabilizes inversion and likelihood optimization while also restricting approximation of sharply concentrated or highly separated targets. The likelihood objective follows the usual change-of-variables formula, \[ \log p_X(x)=\log p_Z(f(x))+\log |\det J_f(x)|, \] with $f$ the transport to latent space [2010.01155, 2102.06539]. In a bi-Lipschitz flow, the log-determinant term is automatically bounded: \[ d\log m \le \log |\det J_f(x)| \le d\log L \] [2010.01155, 2102.06539]. This mitigates numerical instability during maximum-likelihood training. A related theoretical result states that for given base and target distributions, the Jacobian determinant mapping is unique almost everywhere, and for the Quasi-Linear Flow class the globally optimal value of the log-likelihood can be written explicitly in terms of eigenvalues of the pointwise auto-correlation matrix [2102.06539]. That result concerns likelihood structure rather than expressivity per se, but it sharpens the role of determinant control in flow design. Total variation distance is a preferred metric in the expressivity analysis of uniformly bi-Lipschitz flows. For distributions $P,Q$ with densities $p,q$, \[ TV(P,Q)=\frac12\int |p(x)-q(x)|\,dx = \sup_A |P(A)-Q(A)|. \] Pinsker’s inequality gives $2\,TV(P,Q)^2 \le KL(P\|Q)$, so lower bounds in total variation immediately imply nontrivial Kullback–Leibler lower bounds [2107.07232]. The same paper notes that precision/recall style support-based analyses become trivial when both supports are full-dimensional, which motivates TV-based lower bounds. ## 2. Why bi-Lipschitzness is pervasive in flow architectures Many widely used architectures enforce or approximately enforce bi-Lipschitz behavior. In invertible residual networks and Residual Flows, blocks are of the form $f_i=I+g_i$ with $\mathrm{Lip}(g_i)\le L<1$, which implies for $m$ blocks \[ \mathrm{Lip}(F)\le (1+L)^m, \qquad \mathrm{Lip}(F^{-1})\le (1-L)^{-m} \] [2107.07232]. The same scaling appears in practical parameterizations of iResNets, where residual contraction is enforced by spectral normalization [2605.06172]. Glow-like models bound forward and inverse Lipschitz constants through products of operator norms of invertible linear transforms and their inverses [2107.07232]. In affine coupling layers, the Jacobian is triangular, and \[ \log |\det J_f(x)| = \sum_{i\in B} s_i(x_A), \] so bounded scale outputs keep the layer bi-Lipschitz and the log-determinant bounded [2010.01155]. One practical prescription is to bound $s$ so that $\exp(s)\in[e^{-\alpha},e^\alpha]$, thereby keeping singular values of the triangular Jacobian in the same interval [2010.01155]. Spectral normalization, weight clipping, and related operator-norm constraints are standard tools for controlling these constants during training [2107.07232, 2605.06172]. This architectural ubiquity arises from a basic trade-off. Small per-layer Lipschitz constants stabilize inverse evaluation, control Jacobian singular values, and prevent exploding determinants, but they also restrict how much a model can locally expand or compress probability mass [2107.07232]. The literature repeatedly frames this as a conditioning–expressivity tension. The depth and conditioning analysis of affine couplings shows that shallow universality in Wasserstein distance is possible only when ill-conditioning is allowed; as approximation error is driven toward zero in the constructive result, the condition number blows up at least as fast as $1/\epsilon$, so strict bi-Lipschitz control is violated [2010.01155]. Conversely, under bounded Lipschitzness and limited layer width, depth lower bounds appear for approximating complex distributions [2010.01155]. A notable architectural response is the monotone-operator formulation. “Monotone Flows” parameterize a 1-Lipschitz Cayley operator $G$ with $\mathrm{Lip}(G)=L<1$ and define \[ F(x)=((Id+G)/2)^{-1}(x)-x. \] The induced block is globally invertible, $\eta$-strongly monotone, and $\nu$-Lipschitz with \[ \eta=\frac{1-L}{1+L}, \qquad \nu=\frac{1+L}{1-L}, \] so $F$ is bi-Lipschitz with explicit forward and inverse bounds [2210.08176]. This construction is designed precisely to enlarge expressivity relative to contraction-based residual flows while keeping explicit global conditioning control. ## 3. Expressivity limits under uniform bi-Lipschitz constraints The most direct negative theory studies approximation from a standard Gaussian base under a uniformly bi-Lipschitz transport. Several lower bounds in total variation identify target distributions that are intrinsically hard to approximate when $L_1$ and $L_2$ are fixed [2107.07232]. A general dense-subset bound states that if $F$ is $L_1$-Lipschitz and $\eta_A=P^*(A)/\mathrm{vol}(A)$ is the average target density on a measurable set $A\subset\mathbb{R}^d$, then \[ D(P^*,\hat P)\ge \sup_A \mathrm{vol}(A)\Big[\eta_A-\Big(\frac{L_1}{\sqrt{2\pi}}\Big)^d\Big]. \] Thus if some region has average density exceeding $(L_1/\sqrt{2\pi})^d$, the total variation distance cannot vanish [2107.07232]. This formalizes the intuition that a uniformly $L_1$-Lipschitz map cannot create arbitrarily dense regions from a standard Gaussian base. A more geometric dense-ball version uses the fact that an $L_1$-Lipschitz map sends a ball $B_{R,x_0}$ into a latent ball of radius at most $L_1R$. The resulting tight closed-form lower bound is \[ D(P^*,\hat P)\ge \sup_{R,x_0} \Big[ P^*(B_{R,x_0})-\frac{\gamma(d/2,L_1^2R^2/2)}{\Gamma(d/2)} \Big], \] where $\gamma$ is the lower incomplete gamma function [2107.07232]. A looser but dimension-uniform version replaces the Gaussian mass term with $(L_1R)/\sqrt{\pi}$. The complementary sparse-ball bound uses the inverse Lipschitz constant. If $F^{-1}$ is $L_2$-Lipschitz, then around the special point $F^{-1}(0)$, \[ D(P^*,\hat P)\ge \sup_{R>0} \Big[ \frac{\gamma(d/2,R^2/(2L_2^2))}{\Gamma(d/2)} - P^*(B_{R,F^{-1}(0)}) \Big]. \] This states that the model must allocate at least a certain amount of mass near the inverse image of the latent mode, because the inverse cannot compress too aggressively there [2107.07232]. These two mechanisms generate a catalogue of unfavorable target distributions. High-mass spikes, thin ridges, tightly clustered modes, or mass concentrated near low-dimensional manifolds activate the forward-Lipschitz obstruction. Well-separated multimodal mixtures with low-density valleys activate the inverse-Lipschitz obstruction, because the Gaussian base places maximal density near $0$ while the target may place little mass near $F^{-1}(0)$ [2107.07232]. A corollary for two separated target regions $M_1,M_2$ at distance at least $D$ gives \[ D(P^*,\hat P)\ge \frac{\gamma(d/2,D^2/(2L_2^2))}{\Gamma(d/2)}, \] together with an explicit upper bound on maximum precision, \[ \bar\alpha \le 1-\frac{\gamma(d/2,D^2/(2L_2^2))}{\Gamma(d/2)} \] [2107.07232]. Dimension modifies the force of these bounds. For the dense-ball lower bound, the Gaussian mass term $\gamma(d/2,L_1^2R^2/2)/\Gamma(d/2)$ decays rapidly in high dimension unless $L_1R$ is comparable to the typical Gaussian radius, which the paper describes as approximately $\sqrt d$ [2107.07232]. This means that matching a fixed dense ball generally requires larger $L_1$ as dimension grows. By contrast, for the sparse-ball bound, the relevant Gaussian mass decreases with dimension when the effective radius $R/L_2$ is small, so the inverse-Lipschitz limitation is more salient in low dimension for fixed radius ratio [2107.07232]. Empirical examples in the same work illustrate the practical content of these lower bounds. On CIFAR-10, for small per-block $L$ such as $L<0.5$, at least about 10 residual blocks are needed to drive the dense-ball lower bound near zero; for a one-dimensional Gaussian with small variance, at least 7 layers with per-block $L\approx 0.95$ are suggested; on 2D Circles, the inner circle can force the lower bound toward $0.5$; and for separated multimodal targets such as the two-cluster setup and 8-Gaussians, small per-block $L$ may require at least 15 layers to overcome the sparse-ball lower bound in practice [2107.07232]. ## 4. Universal approximation from a diffusion perspective A later line of work argues that the preceding negative results do not show that bi-Lipschitzness itself is an intrinsic barrier. Rather, the obstruction is the imposition of a uniform Lipschitz budget independent of the target accuracy [2605.06172]. The key construction uses the probability flow ODE associated with variance-preserving diffusion. For the constant-rate variance-preserving process, \[ dX_t = -\frac12 X_t\,dt + dW_t, \qquad a(t)=e^{-t/2}, \qquad \sigma^2(t)=1-e^{-t}, \] the associated probability flow ODE is \[ \frac{dx_t}{dt} = -\frac12 x_t - \frac12 s_t(x_t), \qquad s_t(x)=\nabla_x \log p_t(x) \] [2605.06172]. If the score satisfies \[ \sup_x \|\nabla s_t(x)\| \le L(t)<\infty \] for all $t>0$ and $\int_\delta^T (1+L(t))\,dt<\infty$ for every $0<\delta0$, there exists a bi-Lipschitz VP flow $\phi_{0\to T}$ such that \[ \|p_H-(\phi_{0\to T}^{-1})_\# N(0,I)\|_{L^1}<\epsilon \] [2605.06172]. The paper’s interpretation is explicit: bi-Lipschitz constraints per se do not destroy universal approximation power; the negative results arise from demanding uniform bounds that do not scale with the target or the desired error. A plausible implication is that the apparent contradiction between lower bounds and universality is resolved by distinguishing fixed-constant bi-Lipschitz classes from non-uniform unions of such classes. ## 5. Architectural realizations and design strategies Several concrete architectures instantiate bi-Lipschitz normalizing flows with different trade-offs in smoothness, tractability, and expressivity. A first family is based on affine couplings and invertible residual networks. Affine coupling layers preserve triangular Jacobians and analytic log-determinants; the analysis of depth and conditioning shows that constant-depth affine couplings can exactly represent permutations and invertible $1\times1$ convolutions, including the linear maps used in Glow, with $k\le 24$ alternating triangular factors in the theorem statement [2010.01155]. At the same time, at least two full coupling layers are necessary for certain linear maps, and depth lower bounds remain under bounded Lipschitz constraints [2010.01155]. These results imply that partition choice is not the fundamental representational bottleneck, whereas conditioning control still exacts a depth cost. A second family is based on monotone operators. In Monotone Flows, each block is built from the monotone formulation \[ F(x)=((Id+G)/2)^{-1}(x)-x, \] with inverse \[ F^{-1}(y)=((Id-G)/2)^{-1}(y)-y, \] and the implicit relation \[ x-y=G(x+y) \] [2210.08176]. The Jacobian has the closed form \[ J_F=(I+J_G)^{-1}(I-J_G), \] and the log-determinant admits the identity \[ \log\det J_F = \mathrm{tr}\,[\log(I-J_G)-\log(I+J_G)] \] [2210.08176]. Because $\|J_G\|_2\le L<1$, both forward and inverse global Lipschitz constants are known explicitly. The paper combines spectral normalization with the 1-Lipschitz CPila activation, \[ CPila(x)=\alpha_1 [Pila(x-\alpha_2),\,Pila(-x-\alpha_2)]^T, \] with $\alpha_1=1/1.06$, $\alpha_2=0.2$, and $k=5$ in experiments [2210.08176]. A third family emphasizes higher smoothness. Neural diffeomorphic non-uniform B-spline flows construct at least twice continuously differentiable coupling transforms while remaining bi-Lipschitz and retaining analytic inverses for the cubic case [2304.04555]. For a non-uniform B-spline transform of order $k$, the paper gives the sufficient condition \[ \frac{l}{k-1} < \frac{\alpha_j-\alpha_{j-1}}{t_{j+k-1}-t_j} < \frac{u}{k-1}, \] which implies \[ l