Thermodynamic Variational Objectives (TVO)

• Thermodynamic Variational Objectives (TVO) are variational inference bounds derived from thermodynamic integration that unify classical objectives such as ELBO and IW-ELBO.
• TVO constructs a continuous path between an approximate distribution and the target model, enabling tighter lower bounds on log evidence through integration of expectations over intermediary distributions.
• Extensions using weighted Hölder means improve numerical stability and reduce variance, leading to more effective deep generative modeling and inference in complex probabilistic tasks.

Thermodynamic Variational Objectives (TVO) are a class of variational inference (VI) bounds derived from thermodynamic integration, providing a unified and principled framework for tightening and generalizing classical evidence lower bounds (ELBO) in probabilistic modeling. TVOs operate by integrating a path—often the geometric mean—between an approximate variational distribution and the target joint model, yielding a spectrum of variational objectives that subsume ELBO, importance-weighted bounds, Rényi variational inference, and Markov Chain Monte Carlo VI as special cases. Recent developments have extended TVO theory using weighted Hölder means, yielding new exact bounds with superior numerical properties for practical variational inference.

1. Thermodynamic Integration and the Derivation of TVO

Thermodynamic integration formalizes bounds on the partition function difference between two distributions via an integral over a path that continuously interpolates between them. Given two unnormalized densities $\tilde\pi_0(z)$ and $\tilde\pi_1(z)$, with normalizers $Z_0$ and $Z_1$, define

$$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$

for $\beta \in [0,1]$. Thermodynamic integration yields

$$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$

where, in the context of VI, $\tilde\pi_0(z)=q_\phi(z|x)$, $\tilde\pi_1(z)=p_\theta(x,z)$, and $Z_1=p_\theta(x)$, yielding

$\tilde\pi_1(z)$0

Approximating this integral with a left Riemann sum gives the TVO lower bound: $\tilde\pi_1(z)$1 where $\tilde\pi_1(z)$2 (Chen et al., 2021, Masrani et al., 2019).

This construction ensures $\tilde\pi_1(z)$3, where tightness increases with partition count $\tilde\pi_1(z)$4.

2. Exponential Family and Unified Variational Bounds

The path $\tilde\pi_1(z)$5 is a one-dimensional exponential family in $\tilde\pi_1(z)$6, with sufficient statistic $\tilde\pi_1(z)$7. The family takes the form

$\tilde\pi_1(z)$8

with log-partition function $\tilde\pi_1(z)$9. This structure allows direct analysis via Bregman divergences and Taylor remainder theory: $Z_0$0 linking TVO tightness to the geometry of the exponential family path. The TVO formalism unifies several variational objectives:

• ELBO: single left Riemann evaluation at $Z_0$1.
• Importance-Weighted ELBO (IW-ELBO): arises in the $Z_0$2 limit of TVO sums.
• Rényi Variational Inference (RVI): equivalent to partial, rescaled thermodynamic integrals.
• MCMC-VI/Auxiliary-variable VI: recovers the TVO sum from discretizations of $Z_0$3 (Chen et al., 2021, Brekelmans et al., 2020, Brekelmans et al., 2020).

3. Numerical Estimation: Schedules, Estimators, and Gradient Methods

Accurate and efficient estimation of TVO depends on:

• Partitioning (“Schedule”): Placing $Z_0$4 where $Z_0$5 (the integrand) changes most rapidly minimizes Riemann bias. Grid search, moment parameter spacing (equal $Z_0$6 increments), or adaptive schedules via Gaussian process bandit optimization are prevailing approaches, allowing finer grids where curvature or variance is high (Nguyen et al., 2020, Brekelmans et al., 2020).
• Monte Carlo Estimation: Importance sampling under $Z_0$7 with $Z_0$8 samples and normalized weights $Z_0$9 is standard. Reusing base samples across all $Z_1$0 exploits common random numbers for variance reduction.
• Gradient Estimation: The covariance-gradient estimator

$Z_1$1

requires no reparameterization and is applicable to both continuous and discrete latent spaces. For the latent parameter $Z_1$2, a doubly-reparameterized estimator further reduces variance, leading to stable training even for large $Z_1$3 (Masrani et al., 2019, Brekelmans et al., 2020).

4. Geometric Pathologies and the Hölder Bounds Solution

Empirically, the standard geometric path TVO integrand $Z_1$4 exhibits sharp curvature, particularly for $Z_1$5 and $Z_1$6, resulting in high estimator variance and inefficiency. This motivates generalizing the integration path:

• Weighted Hölder Mean Path: Define

$Z_1$7

with $Z_1$8 yielding the geometric mean (TVO) and $Z_1$9 the arithmetic mean.

• Hölder Path and Bounds: The path $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$0 yields local evidence

$$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$1

which is monotonic for $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$2 or $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$3. For suitable $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$4, $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$5 can be made nearly flat, minimizing Riemann sum bias.

• Hölder Bounds (HBO): The objective

$$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$6

is exactly $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$7 for any $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$8. Discretization with near-flat $$\tilde\pi_\beta(z) = p_\theta(x,z)^\beta\, q_\phi(z|x)^{1-\beta}$$9 enables tight, low-variance approximations—sometimes with a single partition (Chen et al., 2021).

5. Practical Algorithms and Empirical Findings

Estimators and Tuning: The importance-weighted estimator for HBO samples $\beta \in [0,1]$0, computes $\beta \in [0,1]$1, builds unnormalized weights $\beta \in [0,1]$2, and normalizes to estimate $\beta \in [0,1]$3. Optimal $\beta \in [0,1]$4 is found via grid or binary search to flatten $\beta \in [0,1]$5 as much as possible (Chen et al., 2021).

Empirical Results:

• On both synthetic and real-world datasets, HBO bounds (with optimal $\beta \in [0,1]$6) approach $\beta \in [0,1]$7 in dramatically fewer partitions than TVO.
• Monte Carlo variance and effective sample size (ESS) are significantly improved (especially for low $\beta \in [0,1]$8).
• In complex posterior inference and generative modeling (e.g., MNIST, Omniglot), HBO yields model and inference network learning superior to TVO, IW-ELBO, and ELBO. For instance, on MNIST, test lower bound (in nats) improves monotonically from ELBO (−94.0) $\beta \in [0,1]$9 IW-ELBO (−88.3) $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$0 HBO (−87.8) (Chen et al., 2021).

6. Theoretical Properties and Generalizations

Tightness and Generality: For geometric TVO, $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$1 is non-decreasing, ensuring $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$2. For the Hölder path, monotonicity holds for $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$3 or $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$4, and the HBO is exact for all $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$5 (Chen et al., 2021).

Exponential Family and Duality: The geometric path $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$6 is an exponential family in $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$7, unifying TVO, rate-distortion, and information bottleneck objectives. The TVO bound gap is a sum of KL divergences between the chain of intermediate exponential family distributions (Brekelmans et al., 2020).

Connection to Hypothesis Testing: The minimizer of the linear combination $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$8 among densities $$\log\frac{Z_1}{Z_0} = \int_0^1 \frac{d}{d\beta}\log Z_\beta\,d\beta = \int_0^1 \mathbb{E}_{Z\sim\pi_\beta}\left[\partial_\beta\log\tilde\pi_\beta(Z)\right] d\beta\;,$$9 is the exponential family member $\tilde\pi_0(z)=q_\phi(z|x)$0. Large deviations arguments identify the same family as optimal for Neyman–Pearson testing, and the Chernoff information is characterized at the $\tilde\pi_0(z)=q_\phi(z|x)$1 matching KL divergences from either endpoint (Brekelmans et al., 2020).

7. Applications and Extensions

TVO and its generalizations have been successfully deployed in:

• Deep Generative Modeling: Discrete and continuous latent variable models, such as VAEs and Sigmoid Belief Networks, with state-of-the-art performance.
• Rate-Distortion and Information Bottleneck: Rate-distortion curves and IB objectives admit an identical path sampling and integration interpretation (Brekelmans et al., 2020).
• Optimization of Schedules: Automatic partition point selection via Gaussian process bandits or moment-parameter adaptive methods yields tighter bounds and improved learning efficiency (Nguyen et al., 2020, Brekelmans et al., 2020, Chen et al., 2021).

Further generalizations exploit the exponential family structure to construct new variational objectives, variational representations, and connections to classical inference and learning problems.

---

References:

• (Chen et al., 2021) Variational Inference with Holder Bounds
• (Masrani et al., 2019) The Thermodynamic Variational Objective
• (Nguyen et al., 2020) Gaussian Process Bandit Optimization of the Thermodynamic Variational Objective
• (Brekelmans et al., 2020) All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
• (Brekelmans et al., 2020) Likelihood Ratio Exponential Families