---
title: Nonconjugate Variational Message Passing
url: https://www.emergentmind.com/topics/nonconjugate-variational-message-passing
type: topic
---

# Nonconjugate Variational Message Passing

Nonconjugate variational message passing (NCVMP) is a class of algorithms extending variational message passing (VMP) to Bayesian models with nonconjugate likelihoods or prior–likelihood pairs. These methods perform approximate posterior inference using exponential-family variational approximations, even when the classical conjugate relationships required for closed-form variational Bayes updates are absent. NCVMP encompasses the key principles of block-coordinate natural-gradient optimization, flexible factor graph representations, and numerical quadrature for handling intractable expected sufficient statistics. This framework has facilitated scalable and accurate variational inference for hierarchical models such as generalized linear mixed models (GLMMs), regression models with non-differentiable losses, and structured prediction architectures.

## 1. Model Setting and Factor Graph Representation

NCVMP is formulated for Bayesian hierarchical models with potentially nonconjugate losses. Consider observations \(y=(y_1,\dots,y_n)\) modeled by a “pseudo-likelihood” defined via a loss function \(L(y_i,\eta_i)\), where the linear predictor is
\[
\eta_i = x_i^\top\beta + z_i^\top u
\]
with fixed effects \(\beta \in \mathbb{R}^p\) and random effects \(u = (u_1, \dots, u_H)\), \(u_h \in \mathbb{R}^{d_h}\). The loss-driven likelihood takes the form
\[
\log\pi(y \mid \beta, u, \sigma^2)
= -\frac{1}{\phi} \sum_{i=1}^n L(y_i, \eta_i)
\]
where \(\phi > 0\). \(L\) may be non-differentiable (e.g., quantile, hinge losses). Priors are typically chosen from conditionally conjugate families, such as:
\[
\beta \sim \mathcal{N}(0, \sigma_\beta^2 R_\beta^{-1}),\quad
u_h | \sigma_h^2 \sim \mathcal{N}(0, \sigma_h^2 R_h^{-1}),\quad
\sigma_h^2 \sim \operatorname{IG}(A_h, B_h),\quad
\sigma^2 \sim \operatorname{IG}(A_0, B_0)
\]
The joint posterior factors naturally into a product of prior and likelihood terms; this permits construction of the corresponding factor graph, with factor nodes for data likelihoods (\(f_i\)), priors, and hyperpriors, and variable nodes for each model parameter [2206.09444].

## 2. Variational Family and Objective Function

NCVMP adopts a mean-field product-form exponential-family variational approximation:
\[
q(\beta, u, \sigma^2, \{\sigma_h^2\}) = q(\beta, u)\prod_{h=1}^H q(\sigma_h^2) q(\sigma^2)
\]
where
\[
q(\beta, u) = \mathcal{N}(m, V),\quad
q(\sigma_h^2) = \operatorname{IG}(\alpha_h, \beta_h),\quad
q(\sigma^2) = \operatorname{IG}(\alpha_0, \beta_0)
\]
The goal is to minimize Kullback–Leibler divergence \(\mathrm{KL}(q \| \pi)\), equivalent to maximizing the evidence lower bound (ELBO):
\[
\operatorname{ELBO}(q) = \mathbb{E}_q [\log \pi(y, \beta, u, \sigma^2, \{\sigma_h^2\})] - \mathbb{E}_q [\log q(\cdot)]
\]
NCVMP performs block-coordinate fixed-point updates of the canonical parameters, shown to correspond to natural-gradient ascent in the space of exponential-family parameters:
\[
\theta_x^{\text{new}} = \mathcal{I}_x^{-1}\nabla_{\theta_x} \mathbb{E}_q [\log \pi(y, \cdot)]
\]
where \(\mathcal{I}_x\) is the Fisher information of \(q(x)\) [2206.09444, 1208.4949].

## 3. Message Passing and Handling Nonconjugacy

In NCVMP, each factor sends a message to variable \(x\) based on the expected log-factor under the variational distribution of other variables:
\[
m_{f \to x}(x) \propto \exp \left\{ \mathbb{E}_{q(\text{others})}[\log f(x, \dots)] \right\}
\]
Conjugate factors (Gaussian, Gamma, etc.) yield closed-form messages corresponding to canonical-parameter contributions.

For nonconjugate data factors—such as those arising from non-differentiable or non-analytic loss functions—the message into the Gaussian block (\(\beta, u\)) involves the expectation of the loss under the variational Gaussian marginal for the linear predictor. For each data unit,
\[
m_i(\beta, u) \propto \exp \left\{ - \frac{\mathbb{E}_q \left[ L(y_i, \eta_i) \right]}{\phi} \right\}
\]
where \(\eta_i \sim \mathcal{N}(m_i, \nu_i^2)\) under \(q(\beta, u)\). The required calculations are summarized by “smoothed-loss” integrals:
\[
\Psi_r(y_i; m_i, \nu_i) = \frac{\partial^r}{\partial m_i^r} \int L(y_i, \eta) \, \mathcal{N}(\eta \mid m_i, \nu_i^2) \, d\eta,\quad r = 0, 1, 2
\]
These yield the expectation, gradient, and Hessian contributions needed for quadratic updates:
\[
\mathbb{E}_q[L(y_i, \eta_i)] = \Psi_{0, i},\quad
\frac{\partial}{\partial m}\mathbb{E}[L] = c_i \, \Psi_{1,i},\quad
\frac{\partial^2}{\partial m\,\partial m^\top}\mathbb{E}[L] = c_i c_i^\top \Psi_{2, i}
\]
If \(L\) is non-differentiable, the existence of \(\Psi_r\) is established via expectations of weak derivatives, and closed-form formulae may be available (e.g., quantile, hinge losses) [2206.09444, 1208.4949, 1205.3906]. Otherwise, one-dimensional quadrature (Gauss–Hermite, Clenshaw–Curtis) is used.

## 4. Algorithmic Implementation: Batch and Stochastic NCVMP

The batch NCVMP algorithm iteratively:
- Computes all \(\Psi_{0,i}, \Psi_{1,i}, \Psi_{2,i}\) for \(i=1, \dots, n\)
- Updates dispersion and random-effect scale parameters via closed-form
- Forms prior precision matrices
- Calculates the natural-gradient Gaussian updates for mean and covariance
- Optionally recomputes ELBO and checks convergence

Explicit update equations for the block parameters are provided. For the Gaussian block,
\[
g = -\bar{R} m - \frac{\gamma_0}{\phi} \sum_{i=1}^n c_i \Psi_{1,i}
\]
\[
H = -\bar{R} - \frac{\gamma_0}{\phi} \sum_{i=1}^n c_i c_i^\top \Psi_{2,i}
\]
\[
V = -H^{-1},\quad m = m - H^{-1}g
\]
Stochastic NCVMP scales to large datasets by sampling mini-batches of size \(s\), estimating \(\hat{\Psi}_{r,i}\), and replacing sums over \(n\) by scaled sums over the batch. Robbins–Monro step sizes update the canonical parameters. Other blocks (inverse-gamma factors) adopt similar stochastic updates [2206.09444, 1208.4949].

## 5. Computational Complexity and Practical Considerations

- Batch NCVMP: Per iteration computational cost is \(O(n\,d_*^2 + d_*^3)\), dominated by quadrature work \(O(n\,q)\) and matrix calculations.
- Stochastic NCVMP: Complexity is reduced to \(O(s\,d_*^2 + d_*^3)\) per update, with memory proportional to batch size.
- Conjugate VMP avoids quadrature, yielding closed forms but similar dominant cost in matrix operations.
- Standard black-box variational inference requires Monte Carlo in the block dimension, often more expensive and high variance.

Initialization from penalized quasi-likelihood avoids local suboptimality. Damping or fixing tuning parameters (e.g., partial noncentering matrices) can stabilize convergence. The ELBO serves as both a convergence monitor and a marginal-likelihood proxy for model comparison [2206.09444, 1205.3906].

## 6. Examples: Quantile, Logistic, and Mixed Models

In quantile regression (\(\tau \in (0,1)\)), the loss is
\[
L(y, \eta) = |y - \eta| (\tau - I\{y - \eta < 0\})
\]
with closed-form \(\Psi_r\) integrals:
\[
\Psi_0(y, m, \nu) = (y - m)[\Phi((y-m)/\nu) + \tau - 1] + \nu^2 \phi((y-m)/\nu)
\]
\[
\Psi_1(y, m, \nu) = 1 - \tau - \Phi((y-m)/\nu),\quad
\Psi_2(y, m, \nu) = \phi((y-m)/\nu)
\]
Updates then follow through the general NCVMP equations. For logistic regression, the loss \(L(y,\eta) = -y\eta + \log(1 + e^\eta)\) lacks analytic \(\Psi_r\); Gauss–Hermite quadrature is applied. GLMMs with partial noncentering leverage tuning matrices \(W_i\) chosen by local curvature of the likelihood to adapt model representation and accelerate convergence [2206.09444, 1205.3906].

## 7. Extensions and Applications

Stochastic NCVMP integrates natural-gradient steps with mini-batch sampling, permitting data-scaling to massive problems. The same message structure underlies diagnostics for prior–likelihood conflict, yielding computationally efficient p-values for Bayesian model checking [1208.4949]. In hierarchical models such as GLMMs, partial noncentering is shown to adaptively yield improved approximation accuracy and accelerated convergence relative to both centered and noncentered schemes [1205.3906]. The approach is broadly applicable to models with non-analytic, non-differentiable, or nonconjugate losses, encompassing settings such as support vector machines and mixed additive models.

## Table: Summary of NCVMP Features

| Aspect                       | Description                                                  | Source           |
|------------------------------|-------------------------------------------------------------|------------------|
| Variational family           | Mean-field, exponential family (Gaussian, IG)               | [2206.09444]     |
| Nonconjugacy handling        | 1-D quadrature for expected sufficient stats                | [2206.09444]     |
| Optimization method          | Block-coordinate, natural-gradient, fixed-point             | [1208.4949]      |
| Stochastic extension         | Robbins–Monro step size, mini-batch sampling                | [1208.4949]      |
| Application domains          | GLMMs, quantile, logistic, general regression               | [2206.09444]     |
| Computational cost           | \(O(n\,d_*^2 + d_*^3)\); batch/stochastic variants          | [2206.09444]     |

NCVMP thus enables scalable, accurate Bayesian inference in nonconjugate models, with broad applicability and robust computational properties, as supported by empirical studies and theoretical analyses [2206.09444, 1208.4949, 1205.3906].

Source: https://www.emergentmind.com/topics/nonconjugate-variational-message-passing