---
title: Amortized Conjugate Posterior (ACP)
url: https://www.emergentmind.com/topics/amortized-conjugate-posterior-acp
type: topic
---

# Amortized Conjugate Posterior (ACP)

The Amortized Conjugate Posterior (ACP) is a hybrid variational inference technique designed for structured probabilistic models, exemplified by the binary noisy-OR latent variable model. ACP integrates classical conjugate dual bounds on the likelihood with the scalability and efficiency of amortized inference, achieving improved posterior approximation and parameter learning by leveraging both model structure and neural inference networks. Originally introduced by Steinhardt and Miller (2019), ACP directly maximizes the evidence lower bound (ELBO) while encoding inductive structure from the generative process into the variational family, enabling efficient and robust inference even in data-scarce regimes [1906.02428].

## 1. Generative Structure: The Noisy-OR Model

ACP operates on discrete latent variable models such as the noisy-OR, where observed binary variables $x = (x_1, ..., x_D) \in \{0,1\}^D$ are generated by a collection of $K$ independent binary latent causes $z = (z_1, ..., z_K) \in \{0,1\}^K$ together with a fixed leak variable $z_0 \equiv 1$. The prior factorizes as $p(z) = \prod_{k=1}^{K} \mu_k^{z_k} (1-\mu_k)^{1-z_k}$, with $\mu_k$ as Bernoulli parameters. The conditional noisy-OR likelihood is given by
$$
p(x_i = 0 | z) = \exp\left(-\theta_{i0} - \sum_{k=1}^K \theta_{ik}z_k\right),
$$
where $\theta_{ik} = -\log(1 - p_{ik})$ with $p_{ik} = p(x_i=1|z_k=1)$. Thus, the marginal log-likelihood $\log p(x) = \log \sum_z p(x,z)$ is combinatorially intractable for exact inference [1906.02428].

## 2. Classical Conjugate Dual Variational Inference

To circumvent intractability, ACP starts from a classical approach: deriving a tractable upper bound for the problematic log-likelihood terms using Fenchel conjugates. For $x_i=1$, the bound
$$
\log p(x_i=1|z) \leq \psi_i a_i - g(\psi_i)
$$
is employed, with $a_i = \theta_{i0} + \sum_k \theta_{ik} z_k$ and $g$ the Fenchel conjugate of $f(s) = \log(1 - e^{-s})$. For $x_i=0$, the true likelihood is retained. This upper-bounded surrogate joint likelihood $p_{UB}(x|z;\psi)$ yields a tractable factorized variational posterior
$$
q(z|x; \psi) = \prod_{k=1}^K \textrm{Bernoulli}(q_k(x; \theta, \mu, \psi)),
$$
with
$$
q_k(x; \theta, \mu, \psi) = \sigma\left(\sum_{i:x_i=1} \psi_i \theta_{ik} - \sum_{i:x_i=0} \theta_{ik} + \log \frac{\mu_k}{1-\mu_k}\right),
$$
where $\sigma$ is the sigmoid function. The classical conjugate dual inference (CDI) approach optimizes each $\psi_i$ per datapoint by fixed-point iteration to tighten the bound on the marginal likelihood [1906.02428].

## 3. Amortized Conjugate Posterior: Formulation

Instead of relocating the variational parameters $\psi$ for each datapoint, ACP amortizes these parameters across the dataset using a neural network encoder, typically a multilayer perceptron (MLP) with two hidden layers (e.g., 200–400 ReLU units, no dropout). For each input $x$, the encoder produces $\psi(x;\varphi) \in \mathbb{R}^D$, which parameterizes the variational posterior in the analytic form above. This construction yields the amortized conjugate posterior
$$
q_\varphi(z|x) = \prod_{k=1}^K \textrm{Bernoulli}(q_k(x;\theta,\mu,\varphi)),
$$
with $q_k$ as defined above but with amortized $\psi_i(x;\varphi)$. The variational family, structured via conjugate duality, inherits inductive bias from the generative model, yielding enhanced generalization in low-data settings [1906.02428].

## 4. ELBO Objective and Training Procedure

ACP directly maximizes the ELBO:
$$
\mathcal{L}(\varphi,\theta,\mu) = \mathbb{E}_{x \sim \text{data}}\Big[\mathbb{E}_{z \sim q_\varphi(z|x)}[\log p(x|z;\theta)] - \text{KL}[q_\varphi(z|x)\Vert p(z;\mu)]\Big].
$$
Training proceeds via stochastic gradient ascent using Adam. For each batch of $x$, the encoder outputs $\psi$, the variational posteriors $q_k$, and Gumbel-Softmax relaxation is used to enable reparameterization and backpropagation through discrete latents. Monte Carlo estimation (with $L=1$ sample per datapoint sufficing in practice) is used for the positive ELBO terms, while negative and KL terms are computed analytically. Gradients with respect to $\varphi$, $\theta$, $\mu$ are taken, and parameters are updated until convergence [1906.02428].

| Step                  | Method               | Details                                  |
|-----------------------|---------------------|-------------------------------------------|
| Encoder               | MLP                 | 2×200 ReLU, linear output, no dropout     |
| Latent variable       | Gumbel-Softmax      | Annealed temperature 1.0→0.1 over 10k steps |
| Optimizer             | Adam                | Learning rate $10^{-3}$, batch size 200   |

## 5. Empirical Benchmarks and Comparative Analysis

Empirical studies across inference accuracy, parameter recovery, generative modeling, and real-world topic modeling evidence that ACP consistently matches or outperforms traditional amortized variational inference (AVI) and unconstrained stochastic variational inference (SVI). With $N_\text{train}=1000$, both ACP and AVI achieve high F1 ($\sim 94\%$), with negative ELBOs of $14.4$ (ACP) and $14.0$ (AVI); for $N_\text{train}=20$, ACP vastly outperforms AVI ($\text{NELBO}=22.2$ vs $37.2$, F1 $76.1\%$ vs $49.2\%$). For generative and parameter estimation tasks, ACP displays superior robustness and generalizes better, particularly in data-scarce regimes where AVI suffers from overfitting and SVI underperforms. In topic modeling of NeurIPS title data, ACP maintains high topic coherence (PMI $2.75$) even with fewer documents, whereas AVI’s coherence drops to $2.55$ [1906.02428].

## 6. Theoretical and Algorithmic Insights

ACP’s hybrid variational family—plugging analytic, model-derived posterior forms into a neural amortization architecture—injects generative structure into encoder learning, yielding posteriors aligned to the model’s statistical dependencies. This structure reduces the risk of overfitting, particularly when the number of examples is limited. A crucial observation is that maximizing tightness of the classical dual likelihood bound (as in CDI) does not ensure optimal ELBO or posterior quality, whereas ACP maximizes the ELBO directly without this misalignment. When sufficient data are available, ACP achieves theoretical parity with unconstrained AVI in flexibility, but its structural bias is advantageous in practical scenarios with limited data [1906.02428].

## 7. Extensions and Related Approaches

The ACP framework generalizes beyond the noisy-OR model to any latent variable model where conjugate dual upper bounds and tractable analytic posteriors can be derived. Related approaches, such as amortized Bayesian inference for clustering models [1811.09747], also exploit the structure afforded by conjugacy within mixture models and Dirichlet process mixtures, using permutation-invariant neural architectures. These methods highlight a growing trend toward integrating analytic structure from classical statistics with the scalability and expressivity conferred by neural amortization. A plausible implication is that amortized conjugate posteriors enable efficient, i.i.d. approximate-posterior sampling at a computational cost competitive with classical MCMC, while capturing structural inductive biases [1906.02428, 1811.09747].

Source: https://www.emergentmind.com/topics/amortized-conjugate-posterior-acp