---
title: Conjugate-Computation Variational Inference (CVI)
url: https://www.emergentmind.com/topics/conjugate-computation-variational-inference-cvi
type: topic
---

# Conjugate-Computation Variational Inference (CVI)

Conjugate-Computation Variational Inference (CVI) is a unified framework for variational inference (VI) in probabilistic models containing both conjugate and non-conjugate structure. CVI leverages closed-form updates for conjugate components while applying stochastic-gradient or surrogate-expansion techniques where conjugacy is absent, yielding an inference method with broad applicability and significant computational advantages in mixed-compositional models. By situating each update as mirror descent in the mean-parameter space of the variational family, CVI balances efficiency and generality, and is directly connected to natural-gradient methods, classical variational message-passing, and online Bayesian filtering algorithms [1703.04265][1712.01038].

## 1. Problem Formulation and Foundations

The central variational problem begins with the observed data $y$ and a model whose joint density can be factorized as
$$p(y, z) = p_{nc}(y, z)\,p_c(y, z),$$
where $z$ denotes latent variables; $p_c(y, z)$ contains terms conjugate to the approximating exponential family, and $p_{nc}(y,z)$ encompasses non-conjugate terms [1703.04265]. The variational approximation is of exponential-family form,
$$q(z|\lambda) = h(z) \exp\{\langle \phi(z), \lambda \rangle - A(\lambda)\},$$
with sufficient statistics $\phi(z)$, natural parameters $\lambda$, log partition function $A$, and base measure $h(z)$.
A bijection exists between natural parameters $\lambda$ and mean parameters $\mu = \mathbb{E}_q[\phi(z)]$. Optimizing the evidence lower bound (ELBO)
$$\mathcal{L}(\lambda) = \mathbb{E}_q[\log p(y, z)] - \mathbb{E}_q[\log q(z|\lambda)]$$
can be equivalently reparameterized as $\widetilde{\mathcal{L}}(\mu)=\mathcal{L}(\lambda(\mu))$ and optimized over $\mu$.

For fully conjugate models, classical variational message passing or coordinate ascent yield closed-form solutions. Non-conjugate factors introduce intractable expectation terms, commonly handled by local analytic bounds or black-box stochastic gradient estimators, at the expense of ignoring remaining conjugate structure or slow convergence [1703.04265].

## 2. CVI Algorithmic Structure and Mean-Parameter Mirror Descent

The core innovation in CVI is the application of mirror-descent in the mean-parameter space rather than stochastic gradient in the natural-parameter space. For mean-parameters $\mu$, the update takes the form
$$\mu_{t+1} = \arg\max_{\mu \in \mathcal{M}} \langle \mu, \widehat{\nabla}_\mu \widetilde{\mathcal{L}}(\mu_t)\rangle - \frac{1}{\beta_t} \mathbb{B}_{A^*}(\mu \|\mu_t),$$
where $A^*$ is the convex dual of $A$, and $\mathbb{B}_{A^*}$ is the associated Bregman divergence. The ELBO gradient decomposes additively:
$$\nabla_\mu \widetilde{\mathcal{L}}(\mu) = \nabla_\mu\mathbb{E}_q[\log p_{nc}] + \nabla_\mu\mathbb{E}_q[\log(p_c/q)].$$

By construction, the non-conjugate gradient $\nabla_\mu \mathbb{E}_q[\log p_{nc}]$ will require Monte Carlo or surrogate evaluation, while the conjugate part yields closed-form natural-parameter updates. The resulting update for the variational distribution is
$$q_{t+1}(z) \propto \exp\{\langle \phi(z), \widetilde{\lambda}_t \rangle\}p_c(y, z),$$
with
$$\widetilde{\lambda}_t = \lambda(\mu_t) + g_t, \quad g_t \approx \nabla_\mu \mathbb{E}_q[\log p_{nc}].$$
Here, $g_t$ can be updated as a running average using the chosen step-size $\beta_t$ [1703.04265][1712.01038].

## 3. Implementation Workflow and Algorithmic Details

The principal algorithmic form for single-factor exponential families is:

| Step | Operation |
|------|--------------------------------------------------------------------------|
| 1    | Compute mean params: $\mu_t = \nabla A(\lambda_t)$                       |
| 2    | Estimate $g_t \approx \nabla_\mu \mathbb{E}_q[\log p_{nc}]_{|\mu_t}$    |
| 3    | Update site: $\widetilde{\lambda}_t = (1-\beta_t)\widetilde{\lambda}_{t-1} + \beta_t g_t$ |
| 4    | Conjugate update: $\lambda_{t+1} = \widetilde{\lambda}_t + \lambda_{\mathrm{prior}}$         |

In mean-field Bayesian networks, coordinate-wise mirror descent across factors recovers variational message-passing (VMP) or stochastic variational inference (SVI) for conjugate terms, with stochastic-gradient corrections for the non-conjugate terms.

Common step-size schedules include $\beta_t = w/(w + t)$ with $w\approx100$, or small constant values, and are not heavily tuned in practice. Computational cost is dominated by the non-conjugate gradient estimation and closed-form conjugate updates, which often scale linearly in data size [1703.04265].

## 4. Theoretical Properties and Convergence

CVI's convergence relies on standard mirror descent results. Provided $\widetilde{\mathcal{L}}(\mu)$ is differentiable with Lipschitz-continuous gradient, stochastic gradient estimates $g_t$ are unbiased and have bounded variance, and the dual log-partition $A^*$ is strongly convex, the iterates converge almost surely to a stationary point of the ELBO. Robbins–Monro conditions on step-sizes are sufficient:
$$\sum_{t}\beta_t = \infty, \quad \sum_{t}\beta_t^2 < \infty.$$
This covers models in which the exponential-family structure is preserved in the variational and conjugate components, subsuming earlier algorithms including non-conjugate VMP with exact gradients and unit step-size [1703.04265].

## 5. Applications and Model Classes

CVI is broadly applicable to any probabilistic model decomposable into conjugate ($p_c$) and non-conjugate ($p_{nc}$) blocks. Covered models include:

- Gaussian-process classification,
- Generalized linear models,
- Kalman filters with non-Gaussian observations,
- Gamma-factor models,
- Poisson–gamma matrix factorization,
- Deep exponential-family models [1703.04265].

For logistic regression, CVI encompasses both Jaakkola–Jordan surrogate bounding and Pólya–gamma data augmentation, rendering logistic likelihoods conditionally conjugate to Gaussian priors. The optimal $q(\beta)$ in binary logistic models is Gaussian with natural parameter updates constructed from closed-form surrogate factors. Iterative updates alternate between closed-form Gaussian updates and auxiliary parameter optimization. In both quadratic bounding and Pólya–gamma augmentation, global parameter updates are analytically tractable, and auxiliary local updates are simple scalar functions of the second moment $E_q[(x_i^\top \beta)^2]$ [1711.06999].

Empirical results demonstrate CVI outperforms fully black-box stochastic-gradient methods in convergence speed and achieves at least comparable predictive performance, with improvements from one to two orders of magnitude in wall-clock time [1703.04265].

## 6. Special Cases, Variants, and Theoretical Connections

CVI generalizes and unifies several existing lines in VI and Bayesian optimization:

- **Gaussian special case:** For $q(\theta) = \mathcal{N}(\mu, \Sigma)$, the updates mirror online Newton's method and the natural-gradient variant of VI, with extensions to diagonal precision (the "Vprop" algorithm). This can be implemented within RMSprop with minimal modification and admits a connection to extended Kalman filtering and regularized natural-gradient descent [1712.01038].
- **Coordinate-ascent VI:** For fully-conjugate models, CVI reduces to classical coordinate-ascent updates.
- **Natural-gradient and online filtering view:** The sequence of CVI updates aligns algebraically with natural-gradient methods under the Gauss–Newton approximation and with Bayesian state filtering in online settings.

## 7. Practical Considerations and Guidance

Implementing CVI requires three key components:

- Efficient computation of natural-to-mean parameter mappings in the chosen exponential family,
- Unbiased Monte Carlo or analytic estimation of $\nabla_\mu \mathbb{E}_q[\log p_{nc}]$,
- The ability to sum natural parameters for conjugate-exponential family inference.

CVI strongly exploits closed-form updates wherever conjugacy is present, with negligible additional overhead in models with partial conjugacy. The method is robust to step-size choice, tolerant to minor tuning, and conceptually modular. In practice, the benefit is most pronounced in large-scale or structured models with a mix of conjugate and non-conjugate factors.

By reframing non-conjugate VI as conjugate VI in an augmented or surrogate-exponential family, CVI recovers efficient variational inference machinery without sacrificing generality or convergence guarantees [1703.04265][1711.06999][1712.01038].

Source: https://www.emergentmind.com/topics/conjugate-computation-variational-inference-cvi