---
title: 'SurVAE Flows: Unifying VAEs and Normalizing Flows'
url: https://www.emergentmind.com/topics/survae-flows
type: topic
---

# SurVAE Flows: Unifying VAEs and Normalizing Flows

SurVAE Flows are a modular generative modeling framework that generalizes normalizing flows and variational autoencoders (VAEs) by incorporating surjective, stochastic, and invertible transformations within a single composable architecture. The formalism enables exact likelihood evaluation with layers that may alter the data dimensionality—a property not available in standard normalizing flows—and naturally unifies previously disparate approaches, such as dequantization, augmentation, and symmetrization, within a single likelihood-based paradigm [2007.02731].


## 1. Foundational Principles and Mathematical Formalism

SurVAE Flows are constructed as compositions of three transformation types between random vectors $z \in \mathcal{Z}$ and data $x \in \mathcal{X}$:

- **Bijective (invertible) transformations**: $x = f(z)$, $z = f^{-1}(x)$. Exact data density: $\log p_X(x) = \log p_Z(z) + \log |\det \nabla_x f^{-1}(x)|$, $z = f^{-1}(x)$.
- **Surjective (dimension-altering) transformations**: Either generative surjections (deterministic $z \to x$, stochastic $x \to z$) or inference surjections (deterministic $x \to z$, stochastic or deterministic $z \to x$). For inference surjections with deterministic $T: x \mapsto z$ and $q(z \mid x) = \delta(z-T(x))$,
  $$
  \log p_X(x) = \log p_Z(T(x)) + \log p(x \mid z=T(x)).
  $$
  $p(x \mid z)$ is supported on the fiber $\mathcal{M}(z) = \{x: T(x) = z\}$.
- **Stochastic layers**: $x \sim p(x \mid z)$ with inference $z \sim q(z \mid x)$. The classic (ELBO-based) VAE structure arises as a special case.

SurVAE Flows compute either the exact data log-density or a tractable variational bound, depending on the position and type of stochasticity in the chain [2007.02731, 2311.14412]. The central likelihood identity for a sequence of $T$ transformations is
$$
\log p_X(x) = \log p_Z(z_T) + \sum_{t=1}^T V_t(x_{t-1}, z_t) - \sum_{t=1}^T E_t(x_{t-1}, z_t),
$$
where $V_t$ is a tractable likelihood contribution and $E_t$ is a non-negative variational gap (zero for bijections and inference surjections).


## 2. Relationship with PDF Projection and Normalizing Flows

The SurVAE framework generalizes the PDF projection theorem of Baggenstoss, which constructs maximum entropy densities on the preimage of lower-dimensional features $z = T(x)$ [2311.14412]. In PDF projection, the back-projected density $G(x)$ is defined as
$$
G(x) = J(x) \cdot g(T(x)), \qquad J(x) = \frac{p_{0,x}(x)}{p_{0,z}(T(x))},
$$
where $p_{0,x}$ is a reference prior and $p_{0,z}$ is its push-forward.

Normalizing flows correspond to the case where $T$ is invertible and dimension-preserving. SurVAE Flows extend this to deterministic surjections $T$ (dimension-reducing or more general maps), specifying the fiber density $p(x|z)$ explicitly, rather than deriving it from a reference prior.

Layerwise, the SurVAE approach allows chaining surjective, bijective, and stochastic blocks, with the exact log-likelihood accruing additively as the sum of log-Jacobian determinants or fiber log-densities [2311.14412].


## 3. SurVAE Layers: Archetypes and Compositionality

SurVAE Flows formalize a range of canonical layers, including:

| Layer Type                | Forward (Gen.)         | Inverse (Inf.)        | Likelihood Contribution             |
|---------------------------|------------------------|-----------------------|-------------------------------------|
| Bijection                 | $x = f(z)$             | $z = f^{-1}(x)$       | $\log|\det \nabla f^{-1}(x)|$       |
| Inference Surjection      | $x \sim p(x|z)$        | $z = T(x)$            | $\log p(x|z)$                       |
| Generative Surjection     | $x = f(z)$             | $z \sim q(z|x)$       | $-\log q(z|x)$ (ELBO contribution)  |
| Stochastic                | $x \sim p(x|z)$        | $z \sim q(z|x)$       | ELBO; see [2007.02731]              |

Key instantiations:
- **Dequantization/Rounding**: $x = \lfloor z \rfloor$, $q(z|x)$ uniform or flow-based on $[x, x+1)$.
- **Slicing**: drop dimensions or slice off variables (e.g., as in multi-scale RealNVP); log-likelihood includes auxiliary/reconstruction density.
- **Absolute Value**: $z = |x|$, with $p(x|z)$ defined on fiber $\{\pm z\}$, log-likelihood involves a fiber mass term.
- **Sorting**: $z = \mathrm{sort}(x)$; likelihood includes a term for the permutation index.

This architecture enables seamless composition: bijective and inference-surjective layers preserve an exact likelihood, while only stochastic or generative-surjective layers introduce a variational bound gap [2007.02731].


## 4. Dimension-Altering Flows and Funnel Layers

Dimension reduction or expansion is achieved by inference surjection layers. The "funnel" layer is a canonical SurVAE block that reduces dimension with exact likelihoods [2112.08069]. Typical funnel constructions include:

- **Convolutional funnel**: Partition $x$; perform local linear/diffeomorphic mapping to $z$ (reduced dimension), and model the residual slice $x_{-}$ with a reconstruction density $p_\varphi(x_{-}|z)$.
- **MLP funnel**: Linear/Triangular mappings split as $W = R + W'$; the Jacobian of the invertible subblock is tractable.
- **Generalized surjective mapping**: Any layer with a tractable right-inverse and log-determinant supports SurVAE composition.

Empirically, funnel-equipped SurVAE Flows attain generative and anomaly detection performance close to or exceeding conventional flows, but with dramatically reduced latent space dimensionality. For instance, F-NSF with $1/16$th latent dimensionality on CIFAR-10 achieved bits-per-dim of 1.71 versus 1.70 for a standard NSF, while significantly improving out-of-distribution detection scores [2112.08069].


## 5. Training and Computational Pipeline

The training procedure for SurVAE Flows parallels that of normalizing flows, except that each module computes either a log-Jacobian or a fiber/slice log-density:
```python
for x in batch:
    z = T(x)
    log_likelihood = log_pz(z) + log_p_x_given_z(x, z)
    loss = -mean(log_likelihood)
    # Backpropagate through T, pz, and p_x_given_z
```
For cascaded architectures, layerwise contributions are summed, with gradients flowing unambiguously through all blocks (bijective, surjective, stochastic) [2311.14412]. For deterministic surjections satisfying the right-inverse condition, the resulting model admits exact maximum likelihood training without a variational bound or looseness term.

Key hyperparameters include partitioning fraction (percentage of dimensions reduced per block), the expressivity of the reconstruction density $p_\varphi$, and regularization (e.g., orthogonality of Jacobian sub-blocks) [2112.08069].


## 6. Applications and Theoretical Impact

SurVAE Flows provide a systematic foundation for several previously ad hoc model extensions:
- **Dequantization** (e.g., Flow++): Unified as variational or uniform surjective mappings [2007.02731].
- **Augmented flows/ANF**: Expressed as generative surjections plus slicing.
- **Symmetrization/Sorting**: Surjective layers folding data onto orbits/noises.
- **Funnel-based compression**: State-of-the-art density modeling with reduced representation complexity [2112.08069].
- **Sampling in discrete/combinatorial spaces**: SurVAE Flow–augmented MCMC improves mixing and effective sample size by smooth transport into continuous latent spaces, then mapping back to discrete states, as demonstrated on Ising and quantized logistic regression benchmarks [2102.02374].

The SurVAE Flow formalism thus subsumes and extends standard NFs and VAEs, affording new flexibility in architecture design, expressivity, and tractable inference in both continuous and discrete support domains.


## 7. Limitations and Open Challenges

While SurVAE Flows considerably expand the space of parameterizable likelihood-based generative models, challenging aspects remain:
- Defining tractable, expressive conditional densities $p(x|z)$ for highly complex or nonlinear surjective maps remains nontrivial; traditional PDF projection techniques offer inversion tools not yet fully exploited in SurVAE flows [2311.14412].
- The practical stability and expressiveness of very deep surjective chains have yet to be fully characterized for large-scale, real-world tasks.
- For stochastic/generative-surjection layers, a variational gap is inevitable, mirroring conventional VAE limitations.
- Large-scale empirical benchmarks have yet to systematically survey SurVAE Flows versus deep flows, VAEs, and other likelihood-based models on diverse modalities [2311.14412].

A plausible implication is that continued advances in fiber and slice density estimation, and connections with classical PDF projection, may further improve these architectures' robustness and tractability.

---

**Key literature**: "SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows" [2007.02731], "A Comparison of PDF Projection with Normalizing Flows and SurVAE" [2311.14412], "Funnels: Exact maximum likelihood with dimensionality reduction" [2112.08069], "Sampling in Combinatorial Spaces with SurVAE Flow Augmented MCMC" [2102.02374].

Source: https://www.emergentmind.com/topics/survae-flows