Hierarchical Variational Models (HVMs)

• Hierarchical Variational Models (HVMs) are variational inference methods that introduce additional latent variables to capture complex, multimodal posterior distributions.
• They employ hierarchical ELBOs and auxiliary reverse models to tighten lower bounds and support scalable, black-box stochastic optimization.
• HVMs unify classical Bayesian inference with modern deep generative techniques, enhancing representation learning and computational efficiency in diverse applications.

Hierarchical Variational Models (HVMs) are a class of variational inference methods that extend the flexibility of approximate posterior distributions by introducing additional latent or variational random variables. This hierarchical structure enables HVMs to capture complex dependencies and multimodalities in latent variable models, including nontrivial marginal posteriors that simple mean-field approximations cannot represent. HVMs admit provable lower bounds on marginal likelihood and are mathematically equivalent to models such as Auxiliary Deep Generative Models under mild assumptions (Brümmer, 2016). Their flexibility extends to discrete and continuous latent variables, supporting black-box inference and scalable stochastic optimization (Ranganath et al., 2015).

1. Model Architecture and Mathematical Formulation

At the core, HVMs introduce a hierarchical auxiliary latent variable $h$ into the variational family. The generative model is unaltered:

$p(x,z) = p(x|z)\,p(z)$

The variational approximation is expanded hierarchically:

$$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$

This factorization requires only the ability to evaluate and sample from each conditional factor. The marginal $q(z|x)$ becomes:

$q(z|x) = \int q(h|x)\,q(z|h,x)\,dh$

which is generally intractable, but sampling-based approximations are straightforward.

The hierarchical structure creates implicit mixture or compound variational families: even if $q(z|h,x)$ is unimodal and simple, integrating over $h$ enables the modeling of complex multimodal or heavy-tailed marginals (Brümmer, 2016, Ranganath et al., 2015).

2. Evidence Lower Bound (ELBO) Derivation and Learning

The standard evidence lower bound,

$$\log p(x)\geq \mathbb{E}_{q(z|x)}[\log p(x,z) - \log q(z|x)]$$,

cannot be used directly because $q(z|x)$ is intractable. Instead, HVMs optimize a joint ELBO over $(z,h)$ but require correction to ensure a valid lower bound. The hierarchical ELBO is constructed as:

$p(x,z) = p(x|z)\,p(z)$0

where $p(x,z) = p(x|z)\,p(z)$1 is an auxiliary "reverse" inference model, ideally matching $p(x,z) = p(x|z)\,p(z)$2, and $p(x,z) = p(x|z)\,p(z)$3 are its parameters (Brümmer, 2016).

This bound is provably valid. When $p(x,z) = p(x|z)\,p(z)$4, it collapses to the standard marginal ELBO (Brümmer, 2016); otherwise, it penalizes mismatch via a KL term.

Pseudocode for HVM learning iteratively samples $p(x,z) = p(x|z)\,p(z)$5, evaluates the per-sample ELBO, computes gradients with respect to both generative and variational parameters, and updates using stochastic optimization such as ADAM (Brümmer, 2016, Ranganath et al., 2015).

3. Expressive Power and Related Variational Families

The essential virtue of HVMs is the flexibility of the marginal $p(x,z) = p(x|z)\,p(z)$6. The induced posterior $p(x,z) = p(x|z)\,p(z)$7 can be arbitrarily expressive even if $p(x,z) = p(x|z)\,p(z)$8 and $p(x,z) = p(x|z)\,p(z)$9 are both simple (e.g., diagonal Gaussians), enabling multimodality, heavy tails, and complex structure in the variational family (Brümmer, 2016).

HVMs subsume and unify several other expressive variational family architectures:

• Auxiliary Deep Generative Models: These models enrich the generative model with an auxiliary variable, while HVMs augment the inference side. Both are mathematically equivalent: the term $$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$0 in the HVM ELBO can be interpreted as an explicit auxiliary generative model factor. The two yield identical ELBOs and operational procedures (Brümmer, 2016).
• Mixture and Flow-based Variational Families: Placing a mixture or flow prior on the variational parameters (as with HVMs) admits a complexity in $$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$1, comparable to, and generalizing, direct mixtures or normalizing flows (Ranganath et al., 2015, Sobolev et al., 2019).

Key theoretical properties include that HVM ELBOs are valid variational lower bounds tightened when the auxiliary reverse $$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$2 closely matches the true conditional, and that optimization behavior inherits standard stochastic variational inference convergence properties (Ranganath et al., 2015).

4. Extensions, Limitations, and Advanced Hierarchical Designs

The hierarchical variational principle generalizes to multi-level settings, semi-implicit construction, and complex dependencies:

• Multi-level Hierarchies: Deep hierarchies (e.g., $$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$3-layer latent stacks) are used for structured data (video, sequence models) and for improved representation learning (Wu et al., 2021, Willetts et al., 2020).
• Semi-Implicit Variational Inference (SIVI/HSIVI): These methods extend HVMs with implicit mixing layers, where only sample access is required for some variational variables. Hierarchical SIVI (HSIVI) provides a principled approach to stacking semi-implicit layers, using layerwise ELBO or score-matching objectives (Yu et al., 2023).
• Importance Weighted and Locally-Enhanced Inference: Recent works apply importance weighting to tighten ELBOs for each group of local random variables or throughout the hierarchy, yielding more accurate posterior approximation, especially when combined with subsampling (Geffner et al., 2022, Sobolev et al., 2019).

Limitations include the requirement of auxiliary density evaluation, potential increases in gradient variance, and calibration of auxiliary reverse distributions $$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$4 or inverse models $$q(z,h|x;\varphi) = q(h|x;\varphi_1)\,q(z|h,x;\varphi_2)$$5 in tighter bounds. For discrete auxiliary variables or complex dependencies, variance reduction techniques or partial analytic marginalization may be required (Ranganath et al., 2015, Sobolev et al., 2019).

5. Connections with Classical and Modern Bayesian Inference

HVMs generalize both classical hierarchical Bayesian modeling and modern amortized variational inference:

• In Bayesian hierarchical regression and empirical Bayes, mean-field or structured variational factorization closely follows the posterior factorization of Bayesian graphical models, enabling conjugate closed-form solutions when available, otherwise using black-box optimization (Becker, 2018).
• In deep generative models, hierarchical priors prevent over-regularization and enable the variational code to reflect the data manifold more faithfully (e.g., Taming VAEs with hierarchical priors) (Klushyn et al., 2019).
• For mixed-type or missing data, hierarchical dependencies among latent variables, combined with advanced sampling or HMC-based inference, have demonstrated state-of-the-art results in imputation and active feature acquisition (Peis et al., 2022).

The unification provided by importance weighted HVMs bridges classical auxiliary-variable bounds, semi-implicit variational inference, and more recent doubly-semi-implicit cases. Each is recovered by appropriate specialization of the IWHVI upper/lower bounds and associated auxiliary distributions (Sobolev et al., 2019).

6. Practical Algorithmic Considerations and Empirical Results

HVMs support scalable learning algorithms with computational complexity linear in retained factorization structure and the number of auxiliary variables. Black-box gradient estimators are enabled by reparameterization and stochastic optimization (Ranganath et al., 2015).

Empirical evaluations demonstrate that HVMs:

• Substantially tighten variational bounds compared to standard mean-field or naive VI in structured models, including deep exponential families, variational autoencoders with structured priors, and hierarchical discrete VAEs (Ranganath et al., 2015, Willetts et al., 2020, Wu et al., 2021).
• Enable disentangled representation learning, particularly in sequence domains and deep hierarchies, by structuring the generative process to separate sequence-level and segment-level factors (e.g., FHVAE) (Hsu et al., 2018).
• Outperform classical and MCMC-based inference in terms of computational efficiency and often show superior predictive and generative accuracy (Becker, 2018, Geffner et al., 2022).
• Can be applied effectively to large-scale data, including speech, video, and missing data scenarios, via advanced local and amortized variational families (Wu et al., 2021, Willetts et al., 2020, Peis et al., 2022).

7. Theoretical Guarantees and Mathematical Equivalence

HVMs and their auxiliary- or semi-implicit equivalents admit sharp theoretical guarantees:

• The hierarchical ELBO is a valid lower bound, tightening to the marginal ELBO when the auxiliary reverse matches the true conditional.
• For importance weighted HVMs, the family of lower and upper bounds constructed (e.g., IWHVI, DIWHVI) converge to the true marginal likelihood as the number of samples increases, and sandwich the standard ELBO and SIVI-type bounds as special cases (Brümmer, 2016, Sobolev et al., 2019).
• Mathematical equivalence holds between HVMs and auxiliary deep generative models: reinterpreting auxiliary distributions between inference and generative side yields identical ELBO objectives and optimization (Brümmer, 2016).

This equivalence, coupled with the black-box optimization framework, marks HVMs as a theoretically well-grounded and practical generalization of variational inference methods in modern probabilistic modeling.

---

Principal references:

(Ranganath et al., 2015) Ranganath et al., "Hierarchical Variational Models" (Brümmer, 2016) Brümmer, "Note on the equivalence of hierarchical variational models and auxiliary deep generative models" (Sobolev et al., 2019) Bauer et al., "Importance Weighted Hierarchical Variational Inference" (Geffner et al., 2022) Kingma et al., "Variational Inference with Locally Enhanced Bounds for Hierarchical Models" (Wu et al., 2021, Willetts et al., 2020, Hsu et al., 2018, Klushyn et al., 2019, Becker, 2018) for application- and domain-specific developments.