Auxiliary Variable Sampling for Truncated Likelihoods

• The paper introduces auxiliary variable sampling that reformulates inference for truncated likelihoods by embedding the target in an augmented high-dimensional space.
• It details methods such as piecewise-continuous HMC, Dirichlet–multinomial Gibbs sampling with geometric augmentation, and the Multiple Auxiliary Variable approach.
• The approach significantly improves mixing, scalability, and efficiency in sampling constrained and intractable posterior distributions.

Auxiliary variable sampling for truncated likelihoods encompasses a class of data augmentation and Markov chain Monte Carlo (MCMC) techniques specifically developed to handle probability models in which the likelihood, prior, or posterior incorporates hard constraints, truncations, or normalization intractabilities. By introducing one or more auxiliary or latent variables, these techniques reformulate the sampling task to circumvent nonstandard supports or normalization constants, allowing efficient and theoretically grounded posterior inference. This methodology provides substantial improvements in mixing, tractability, and ease of implementation over conventional approaches such as standard Metropolis–Hastings or Gibbs samplers.

1. Fundamental Principles of Auxiliary Variable Methods

Auxiliary variable methods recast inference problems with truncated or intractable likelihoods into higher-dimensional spaces where standard sampling approaches apply. Given a distribution

$p(x) \propto L(x)\,\mathbf{1}_G(x),$

where $L(x)$ is a smooth density and $\mathbf{1}_G(x)$ encodes a hard constraint region $G$, conventional MCMC is inefficient due to abrupt boundaries and non-differentiability. Auxiliary variables create an augmented space in which either: (a) piecewise-smooth densities are sampled directly via Hamiltonian or event-based dynamics, (b) geometric series expansions neutralize problematic factors, or (c) auxiliary chains and intermediate distributions bypass calculation of intractable normalizing constants.

2. Piecewise-Continuous Augmentation and Exact HMC

For continuous variables subject to hard constraints (e.g., positivity, bounded domains), the piecewise-continuous augmentation strategy embeds the original target in a Hamiltonian system:

$H(x,p) = U(x) + \tfrac{1}{2}p^\top M^{-1}p,$

with

$$U(x) = \begin{cases} -\log L(x), & x \in G \ +\infty, & x \notin G \end{cases}$$

and $p\sim\mathcal{N}(0,M)$. Standard Hamiltonian dynamics are employed inside $G$, while the boundaries $\{g_n(x)=0\}$ are treated as either reflecting or refracting surfaces depending on the magnitude and jump in the potential $U(x)$. If the boundary is a hard truncation (infinite energy jump), a perfectly elastic reflection is enforced by updating the momentum according to the normal at the boundary. If the boundary admits a finite potential jump, energy conservation dictates a possible refractive passage or reflection contingent on whether the normal component of the momentum remains real and non-negative. This construction yields an exact, rejection-free HMC scheme for truncated densities (Pakman et al., 2013).

3. Auxiliary-Variable Data Augmentation for Truncated Multinomials

For models involving discrete latent variables and truncated multinomial likelihoods, the introduction of auxiliary geometric waiting-time variables achieves conjugacy. For instance, in the case of Dirichlet priors conjugate to truncated multinomial likelihoods, introducing additional geometric latent variables $k_{rj}$ expands denominators of the form $(1-\pi_r)^{m_{r\cdot}}$ by their geometric series representation. The joint augmented target becomes amenable to exact Gibbs sampling:

• Auxiliary variables $k_{rj}$ are sampled as independent geometric random variables given the current parameter values.
• Parameters (e.g., $\pi$) are sampled from a Dirichlet distribution with updated counts that include contributions from the auxiliary variables (Johnson et al., 2012). This augmentation yields a two-step alternating Gibbs sampler with improved mixing and scalability compared to generic Metropolis–Hastings, as empirically observed by autocorrelation reduction, faster convergence in potential scale reduction, and accelerated moment statistics convergence.

4. Auxiliary Variable Methods for Intractable and Doubly-Intractable Likelihoods

In undirected graphical models, such as Markov random fields (MRFs) possessing intractable partition functions, auxiliary variable methods circumvent the need to compute normalizing constants directly. The Multiple Auxiliary Variable (MAV) method constructs a reverse-time Markov chain of auxiliary variables bridging a tractable density and the target density via intermediate stages, each associated with reversible kernels and intermediate densities. An unbiased estimator for the reciprocal normalizing constant is constructed via

$$\hat v(\theta) = \prod_{i=2}^N \frac{\gamma_{i-1}(u_i|\theta,y)}{\gamma_i(u_i|\theta,y)},$$

where $\gamma_i$ are unnormalized bridging densities and $u_i$ are the auxiliary variables sampled via MCMC kernels reversible with respect to their target stage density. The overall algorithm operates as a pseudo-marginal Metropolis–Hastings sampler using these unbiased weights, and targets the precise posterior without ever computing $Z(\theta)$, provided that the estimator is unbiased (Prangle et al., 2016). This methodology readily generalizes to both truncated and other intractable likelihood settings with appropriate selection of intermediate densities and transition kernels.

5. Algorithmic Implementations

The following table summarizes key auxiliary variable sampling schemes for truncated likelihoods described in the literature:

Method	Augmentation Type	Notable Applications
Piecewise-continuous HMC	Momentum/cell boundaries	Truncated continuous models, spike-and-slab regression with constraints (Pakman et al., 2013)
Dirichlet–Truncated Multinomial Gibbs	Geometric waiting times	HDP-HSMM, Dirichlet process mixtures (Johnson et al., 2012)
Multiple Auxiliary Variable (MAV)	Bridging densities & chains	MRFs and posteriors with intractable $Z(\theta)$ (Prangle et al., 2016)

In the HMC context, each iteration entails closed-form integration within each cell, exact computation of wall-crossing times, and analytical update of momenta at each boundary encounter. For Dirichlet-multinomial scenarios, exploiting the geometric series expansion guarantees conjugacy within the augmented space, yielding a pure Gibbs procedure with no tuning parameters. Within the MAV method, the number of bridging steps, structure of intermediate densities, and efficiency of transition kernels critically govern variance and per-iteration cost.

6. Computational Properties and Comparative Performance

Auxiliary variable methods for truncated likelihoods typically exchange moderate per-iteration increases in computational or algebraic complexity for substantial gains in mixing rate and exploration efficiency. For example:

• The piecewise-continuous HMC method maintains acceptance probability 1, thereby eliminating the need for finite-difference step size tuning and achieving effective sample sizes per unit CPU time several orders of magnitude greater than coordinate-wise Metropolis or Gibbs when sampling strongly correlated, truncated posteriors.
• The Dirichlet-auxiliary Gibbs sampler achieves uniform ergodicity and low parameter autocorrelation, especially in high-dimensional settings or when the truncated counts are not excessively large.
• MAV offers exactness for doubly-intractable problems subject to unbiasedness of its importance weights, with practical efficiency determined by the variance of the estimator and the choice of bridging densities.

7. Broader Implications and Research Connections

Auxiliary variable techniques for truncated likelihoods have wide-ranging applicability across Bayesian hierarchical modeling, latent structure inference, sparse regression, and undirected graphical model learning. They subsume and extend classical data augmentation, reversible-jump, and pseudo-marginal MCMC techniques. A plausible implication is that this class of methods may serve as a unifying framework for tackling increasingly complex inferential problems involving nonstandard supports, hard constraints, and intractable normalization. Further research directions include fine-tuning bridging density schedules in MAV, optimizing event-finding routines in piecewise HMC, and extending auxiliary-variable approaches to more general conditional or even nonparametric inference tasks (Pakman et al., 2013, Johnson et al., 2012, Prangle et al., 2016).