Hierarchical Variational Auto-Encoders (HVAE)

• HVAE is a generative model that replaces a single latent variable with a hierarchy of variables, each governed by top-down conditional priors.
• The architecture enhances separation between global and local features, facilitating effective modeling of multimodal, sequential, and structured data.
• Training leverages layerwise KL divergence and top-down inference to mitigate posterior collapse and optimize the evidence lower bound (ELBO).

A hierarchical variational auto-encoder (HVAE) is a variational autoencoder in which latent variables are organized in more than one level, so the generative model has conditional structure among latent variables rather than a single flat latent code. In the canonical formulation, hierarchy means a stack of stochastic latent variables with top-down conditional priors and a matching hierarchical approximate posterior; in adjacent literatures, however, the term has also been used for multimodal shared/local factorizations, sequence-level versus segment-level decompositions, graph- and scale-structured latents, and several noncanonical constructions in which “hierarchy” is placed in the input representation, latent geometry, or inference pipeline rather than in latent-variable depth (Hsu et al., 2018, Vahdat et al., 2020, 2502.02856).

1. Canonical probabilistic structure

The defining distinction between a standard VAE and an HVAE is the replacement of a single latent variable $z$ with a hierarchy $z_{1:L}$. A standard VAE is typically written as

$$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$

whereas an HVAE introduces multiple latent variables with conditional dependence. One generic form used in the literature is

$$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$

with the Markov special case

$$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$

The corresponding hierarchical posterior is

$$q(z_{1:L}\mid x)=q(z_L\mid x)\prod_{\ell=1}^{L-1} q(z_\ell\mid z_{>\ell},x).$$

These factorizations are explicit in analyses of hierarchical latent-variable models and in top-down image HVAEs (Zhao et al., 2017, Dang et al., 2023).

This hierarchical organization is often used to align latent variables with different scopes or abstraction levels. In multimodal HVAEs, a top latent can encode shared multimodal structure while lower latents remain modality-specific; in sequential HVAEs, one part of the hierarchy can encode slowly varying sequence-level factors while another captures rapidly varying segment-level content. For example, the multimodal hierarchical VAE factorizes

$$p_\theta(\mathbf{x}_{1:N},\mathbf{z}_{1:N}^m,\mathbf{z}^c) = p(\mathbf{z}^c)\prod_{i=1}^N p_\theta(\mathbf{z}_i^m\mid \mathbf{z}^c)\,p_\theta(\mathbf{x}_i\mid \mathbf{z}_i^m),$$

while the factorized sequential HVAE uses a sequence-level latent mean $\mu_2$, sequence-dependent $z_2^{(n)}$, and segment-level $z_1^{(n)}$ (Vasco et al., 2020, Hsu et al., 2018).

2. Variational inference and training

The training objective remains the ELBO, but in an HVAE the KL term decomposes across latent levels. In top-down hierarchical image VAEs this takes the familiar form

$z_{1:L}$0

with the hierarchical KL written as a sum of layerwise terms such as

$z_{1:L}$1

Operationally, each latent block computes a conditional prior, a conditional posterior, a sample, and a per-layer KL contribution (Child, 2020).

Hierarchical inference is usually not purely bottom-up. In ladder-style and top-down image HVAEs, the posterior at level $z_{1:L}$2 is parameterized from both the current top-down decoder state and the corresponding bottom-up encoder features. NVAE, for example, uses conditional Gaussian priors across latent groups and hierarchical conditional posteriors, together with a residual posterior parameterization

$z_{1:L}$3

and stabilizes training with KL warm-up, KL balancing during warm-up, and spectral regularization (Vahdat et al., 2020).

Training recipes vary across subfamilies. MHVAE uses modality representation dropout to approximate a joint-modality posterior from arbitrary subsets of modalities and reports KL warm-up schedules inspired by Ladder VAE, with $z_{1:L}$4 in the main reported runs (Vasco et al., 2020). Very deep image HVAEs use additional stabilization heuristics; VDVAE reports a KL training schedule in which the posterior is first trained against a standard normal while the prior is trained separately to predict posterior parameters, then switched back to the true posterior-prior KL near the end of training (Child, 2020). IA-HVAE modifies inference rather than the latent hierarchy itself by combining an amortized initialization with iterative refinement using decoder gradients, enabled by a linearly separable decoder in a transform domain, and reports a $z_{1:L}$5 speed-up for iterative inference with respect to a traditional HVAE (Penninga et al., 22 Jan 2026).

3. Major architectural families

A large fraction of the literature uses the canonical top-down latent hierarchy for image density modeling. NVAE is a deep hierarchical VAE with many stochastic groups arranged across multiple spatial scales, hierarchical priors $z_{1:L}$6, hierarchical posteriors $z_{1:L}$7, and a multiscale convolutional decoder. Its reported configurations range from $z_{1:L}$8 latent groups on MNIST to $z_{1:L}$9 groups on CelebA-HQ and FFHQ $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$0, with groups distributed over scales such as $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$1. VDVAE extends the same top-down family to very deep stochastic depth and argues that sufficiently deep VAEs can represent autoregressive models while preserving fast sampling (Vahdat et al., 2020, Child, 2020).

A second family uses hierarchy to separate shared and local factors rather than only coarse and fine visual structure. MHVAE places a shared core latent $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$2 above modality-specific latents $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$3, making cross-modality inference natural. FHVAE places a sequence-level latent mean $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$4 above per-segment $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$5, alongside segment-level $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$6, to disentangle persistent from transient factors in speech and other sequential data. HDUVA replaces an observed domain label with a latent topic variable $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$7 that generates a domain-specific latent $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$8, alongside class-specific $$p(\mathbf{x},\mathbf{z}) = p(\mathbf{z})p(\mathbf{x}\mid \mathbf{z}),$$9 and residual $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$0, thereby using hierarchy to model latent domain structure rather than only reconstruction fidelity (Vasco et al., 2020, Hsu et al., 2018, Sun et al., 2021).

A third family introduces hierarchy over specialized data structures. HG-VAE combines a deep hierarchical latent stack with graph-convolutional networks so that latents $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$1 live on graphs of different size and capture motion at different abstraction levels; the top latent can be represented by a single graph node for global action structure, while lower latents restore local motion detail (Bourached et al., 2021). Relaxed-Responsibility Hierarchical Discrete VAEs build a many-layer discrete HVAE with categorical latent variables parameterized by relaxed responsibilities in embedding space, enabling end-to-end training with up to $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$2 latent layers (Willetts et al., 2020). Greedy HVAEs for video prediction keep a deep hierarchical architecture but train it module by module, freezing lower modules while adding deeper ones, and use only the deepest latent at test time; this is a true hierarchical generative architecture, but not a jointly trained conventional HVAE (Wu et al., 2021).

4. Applications and empirical roles

HVAEs have been used wherever multi-scale or multi-scope latent structure is operationally useful. In multimodal learning, the hierarchy separates shared multimodal content from modality-specific generation. MHVAE is designed to learn modality-specific distributions together with a joint-modality distribution that supports arbitrary-subset inference and cross-modality generation, and is evaluated on bimodal MNIST, FashionMNIST, and CelebA (Vasco et al., 2020).

In speech and sequential acoustics, hierarchy is often matched to temporal persistence. FHVAE learns sequence-level and segment-level factors and has been used for speaker verification, robust speech recognition, and voice conversion. Its scalable hierarchical-sampling training algorithm is evaluated from $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$3 to $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$4 hours and is intended precisely to preserve sequence-scoped latent semantics at scale (Hsu et al., 2018).

In video prediction, hierarchy is used to capture multi-level stochasticity of future observations. GHVAE reports $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$5 gains in prediction performance on four video datasets, a $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$6 higher success rate on real robot tasks, and monotonic improvement as more modules are added (Wu et al., 2021). In human motion, HG-VAE is used to generate coherent actions, detect out-of-distribution data, and impute missing data by gradient ascent on the model’s posterior; on H3.6M and AMASS it is presented as a holistic generative model of action over multiple time-scales (Bourached et al., 2021).

In inverse problems, HVAEs have been repurposed as structured priors. PnP-HVAE uses a deep hierarchical image VAE as a Plug-and-Play prior, yielding a reconstruction operator

$$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$7

with a fixed-point interpretation tied to a tempered HVAE prior; the reported method is competitive with state-of-the-art denoiser-based Plug-and-Play methods and with other restoration methods based on generative models (Prost et al., 2023). In domain generalization and fair representation learning, hierarchy has been used to isolate domain-specific or sensitive information from task-specific content: HDUVA learns class-label-specific and domain-specific latents without domain supervision, while H-VFAE + VP combines a two-layer latent hierarchy, a VampPrior on the top latent, and fairness regularization based on MMD or mutual information (Sun et al., 2021, Botros et al., 2018).

5. Posterior collapse, disentanglement, and theoretical critiques

A persistent theme in the HVAE literature is that deeper hierarchies do not automatically imply effective latent usage. A common belief has been that top-down hierarchical VAEs avoid posterior collapse more effectively than shallow VAEs, but this has been challenged directly. One study of top-down HVAEs reports that on MNIST, as the number of stochastic layers increases from $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$8 to $$p(x,z_1,\ldots,z_L) = p(x\mid z_{>0}) \prod_{\ell=1}^{L-1} p(z_\ell\mid z_{>\ell})\, p(z_L),$$9, active units remain below $$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$0 and about $$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$1 of latent variables have almost zero KL, even though NLL improves; the same work proposes a deterministic top context derived from DCT coefficients to increase latent usage without harming generative performance (Kuzina et al., 2023).

Theoretical analyses sharpen this critique. For a two-level Markovian HVAE with linear-Gaussian components, posterior collapse at the upper latent is shown to depend on the effect of learnable encoder variance: when $$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$2 is learnable and the optimal $$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$3 is low-rank, the corresponding dimensions of the second latent collapse, whereas the lower latent need not collapse in the same way when $$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$4 is unlearnable isotropic (Dang et al., 2023). More generally, a foundational critique of hierarchical latent-variable models argues that, under ideal ELBO optimization, stacked stochastic latent layers need not yield hierarchical feature learning, and that a Gibbs chain using only the bottom latent layer can already reproduce the data distribution in a Markov HVAE; this analysis motivates VLAE, which shifts hierarchy from latent-variable depth into architectural pathway depth (Zhao et al., 2017).

Disentanglement claims in HVAEs are correspondingly heterogeneous. FHVAE uses an explicit sequence/segment factorization and an additional discriminative term over sequence identities because the ELBO alone does not guarantee the intended separation (Hsu et al., 2018). PH-VAE claims “some form of disentangled representation learning ability,” but it provides no standard disentanglement metrics such as MIG, DCI, SAP, FactorVAE score, or beta-VAE metric, and its mutual-information interpretation is explicitly nonstandard (2502.02856). This suggests that HVAE depth, by itself, is neither a sufficient condition for disentanglement nor a substitute for careful analysis of which latent variables remain active.

6. Terminological boundaries and noncanonical uses of “hierarchy”

The term “HVAE” is not used uniformly. In the canonical sense, hierarchy refers to latent-variable depth. In several adjacent works, however, “hierarchy” refers to ordered structure elsewhere in the model.

Meaning of “hierarchy”	Representative paper	Precise status
Stacked latent variables with conditional priors	NVAE (Vahdat et al., 2020)	Canonical HVAE
Shared top latent with lower modality-specific latents	MHVAE (Vasco et al., 2020)	Canonical HVAE
Sequence-level and segment-level latent factors	FHVAE (Hsu et al., 2018)	Canonical HVAE specialized to sequential scope
Polynomial powers of the input with multiple encoder branches	PH-VAE (2502.02856)	Noncanonical; hierarchy is in input representation and encoder branches
Hyperbolic latent geometry for hierarchical data	Poincaré VAE (Mathieu et al., 2019)	Noncanonical; hierarchy is geometric, not architectural
AE front-end followed by VAE for authentication	IIoT HVAE (Meng et al., 9 Aug 2025)	Noncanonical; hierarchy is a cascaded AE+VAE pipeline

These boundary cases matter because they mark different research questions. PH-VAE introduces a hierarchy in polynomially transformed copies of the same observation,

$$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$5

assigns one encoder to each order, averages the branchwise latent statistics, and calls the averaged KL penalty a “Polynomial Divergence”; mathematically, however, this quantity is simply

$$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$6

so the model is best read as a modified VAE with a polynomial multi-branch encoder rather than as a true hierarchical latent-variable HVAE (2502.02856). Poincaré VAE is likewise about hierarchy in the geometry of the latent space and in the data, not in stacked stochastic latent layers; it remains a single-latent-layer VAE with a Poincaré ball latent space (Mathieu et al., 2019). The IIoT “HVAE” is a domain-specific AE+VAE cascade in which an AE reduces CIR dimensionality and a VAE with one unimodal and one bimodal latent branch performs authentication; its “hierarchy” is a serial processing hierarchy, not a classical hierarchical prior over multiple stochastic latent layers (Meng et al., 9 Aug 2025).

The broadest implication is that “hierarchical” in the VAE literature can denote latent depth, scope factorization, graph or scale organization, geometric structure, or cascaded processing. For work specifically on hierarchical variational auto-encoders in the narrow probabilistic sense, the canonical reference point remains the top-down latent hierarchy

$$p(x,z) = p(x\mid z_1)\prod_{\ell=1}^{L-1} p(z_\ell\mid z_{\ell+1})\,p(z_L).$$7

together with the corresponding ELBO decomposition and the associated questions of latent usage, multiscale representation, and optimization stability (Vahdat et al., 2020, Dang et al., 2023).