Invertible Generative Models Overview

• Invertible generative models are probabilistic frameworks that create a bijective mapping between latent variables and data samples, facilitating both forward simulation and inversion.
• The use of composite triangular flows ensures robust conditional sampling for both likelihood simulation and posterior inference in complex inverse problems.
• This approach enables exact likelihood evaluation and efficient density estimation, with applications in areas such as imaging and Bayesian inversion.

Invertible generative models are probabilistic models that provide an invertible, bijective mapping between latent variables (drawn from a known, often simple, base distribution) and data samples. The invertibility ensures that for every data point, there exists a unique latent code, and vice versa. This property is foundational in flow-based and related frameworks for exact likelihood evaluation, bidirectional inference, conditional simulation, and the modeling of both forward and inverse problems. Invertibility is central to efficient density estimation, conditional sampling, and tractable Bayesian inference.

1. Mathematical Structure and Triangular Flows

An invertible generative model defines a mapping $F: \mathbb{R}^d \rightarrow \mathbb{R}^d$ such that both $F$ and $F^{-1}$ exist and are tractable. In the context of forward and inverse problems with unknowns $u \in \mathbb{R}^n$ and measurements $f \in \mathbb{R}^m$, it is often required to efficiently perform conditional sampling, i.e., sample the likelihood $\mu_{F|U=u}$ and the posterior $\mu_{U|F=f}$.

Triangular normalizing flows play a crucial role due to their conditioning properties. A lower-triangular flow $F(x, y) = (F_1(x), F_2(x, y))$ enables one to fix $x$ and transform $y$ so that conditional densities can be sampled by substitution. The corresponding upper-triangular flow $F(x, y) = (F_1(x, y), F_2(y))$ supports the symmetric inversion required for Bayesian inference.

A combined invertible map $S$ over $(u, y)$ is constructed by embedding both a lower- and an upper-triangular flow: $$S_1(u, y) = F_{\text{post}}^{-1}(u; F_{\text{like}}(y; u)), \qquad S_2(u, y) = F_{\text{like}}(y; u)$$ with inverse map $R = S^{-1}$ given by

$$R_1(x, f) = F_{\text{post}}(x; f), \qquad R_2(x, f) = F_{\text{like}}^{-1}(f; F_{\text{post}}(x; f))$$

This composite structure guarantees conditional consistency: sampling $y$ given $u$ via the S mapping pushes forward the base measure to the likelihood, and symmetrically, sampling $x$ given $f$ via $S^{-1}$ produces posterior samples.

2. Algorithmic and Theoretical Properties

The triangular flow composition leverages structural properties of conditional distributions:

• Lower-triangular flows are suited for likelihood simulation since for fixed $u$, the map $F_2$ handles the transformation from the base distribution to the conditional measurement.
• Upper-triangular flows enable posterior sampling since for fixed measurement $f$, the first component $F_1$ conditions on $f$ to generate the unknown $u$.
• Invertibility is preserved globally through careful design: the composed map $S$ and its inverse $R$ are both bijective, satisfying $S^{-1} = R$.

This ensures that forward simulation ($u \rightarrow f$), inverse Bayesian inference ($f \rightarrow u$), and joint density evaluation are all feasible in a single model—a significant advantage over models with only unidirectional conditional tractability.

The conditional sampling mechanisms are underpinned by theoretical results proving that

• $(R_1(\cdot, f))_{\#} \mu_X = \mu_{U|F=f}$
• $(S_2(u, \cdot))_{\#} \mu_Y = \mu_{F|U=u}$

where $\mu_X, \mu_Y$ are reference (typically standard Gaussian) measures.

3. Training and Loss Functions

Model training seeks to align the pushforward of the reference distribution through the invertible map with the target joint distribution $\mu_{U,F}$. The principal loss is derived from the Kullback–Leibler divergence between the pushforward under the learned map and the empirical data distribution: $$\mathbb{E}_{(u, f) \sim \mu_{U, F}}\big[\frac{1}{2}\|F^{-1}(u, f)\|_2^2 - \log|\nabla F^{-1}(u, f)|\big]$$ The triangular structure ensures that the loss separates into contributions for the lower and upper triangular flows: $$\underset{F_{\text{post}}}{\text{min}}\: \mathbb{E}[\frac{1}{2}\|F_{\text{post}}^{-1}(u; f)\|_2^2 - \log|\nabla F_{\text{post}}^{-1}(u; f)|],$$

$$\underset{F_{\text{like}}}{\text{min}}\: \mathbb{E}[\frac{1}{2}\|F_{\text{like}}^{-1}(f; u)\|_2^2 - \log|\nabla F_{\text{like}}^{-1}(f; u)|].$$

Additionally, discrepancy terms such as Sinkhorn or MMD metrics ($\mathcal{M}$) between the model's conditional pushforwards and empirical conditional distributions can be included to enforce conditional correctness.

4. Numerical Examples and Conditioning

The paper presents numerical evaluations in several regimes:

• Linear-Gaussian Settings: When the prior on $u$ and the likelihood are both Gaussian and the forward model is linear, the combined map $S$ is linear and its condition number is analyzed. A notable finding is that integrating both triangular flows into a composite map can improve invertibility under ill-posedness scenarios, particularly in the “noiseless” regime where conventional flows can become ill-conditioned.
• Nonlinear Flows: The approach generalizes to nonlinear triangular flows parameterized, for example, via Hermite polynomials and trained variationally. Visualizations confirm that the learned invertible map appropriately transforms between the standard Gaussian base and the joint distribution, preserving the ability to perform conditional inference and simulation.
• Inpainting and Partial Observations: In applied imaging tasks (such as digit inpainting), the model enables both synthetic data generation and uncertainty-aware estimation of missing content, again leveraging the conditional sampling afforded by the triangular structure.

5. Applications and Implications

The described invertible generative model unifies simulation and inference in Bayesian inverse problems within a single framework. This dual capability is particularly important in domains requiring:

• Efficient forward data simulation for uncertainty quantification, experimental design, and sensitivity analysis.
• Robust Bayesian inversion where accurate posterior quantification (including uncertainty) is critical, especially under partial observation or ill-posed settings.

The triangular flow construction is broadly applicable to high-dimensional and structured inverse problems, in fields ranging from computational imaging to geophysics and engineering.

6. Conditioning, Robustness, and Extensions

A key property of the composite approach is its ability to moderate the ill-conditioning that can affect ordinary triangular flows, especially as measurement noise decreases and the forward operator becomes nearly singular. Analysis of the condition numbers of the various mapping components shows improved numerical stability for the combined invertible map in well-posed regimes.

Extensions to the core concept include:

• Adapting the structure to arbitrary reference measures and target distributions (Gaussian or otherwise).
• Alternate metrics for the loss function, including maximum mean discrepancy and optimal transport losses, for greater flexibility in capturing complex empirical distributions.
• Generalization to non-affine flows for modeling more expressive or highly nonlinear conditional distributions required in real-world applications.

7. Summary Table: Components and Properties

Map Type	Primary Use	Conditional Sampling Target
Lower-Tr. Flow (F₂)	Likelihood simulation	$\mu_{F|U=u}$, fixed $u$
Upper-Tr. Flow (F₁)	Posterior inference	$\mu_{U|F=f}$, fixed $f$
Composite Map (S/R)	Bidirectional (fwd/inv)	Unified simulation & inference

This structured approach—constructing invertible generative models by combining upper and lower triangular flows—yields a flexible, principled solution for forward and inverse problems, ensuring conditional tractability, invertibility, and robust bidirectional inference (Leeuwen et al., 4 Sep 2025).