Alternating Gibbs Sampling Overview

• Alternating Gibbs Sampling (AGS) is a block Gibbs technique that alternates conditional updates between variable groups, ensuring all proposals are accepted without Jacobian calculations.
• AGS streamlines transdimensional Bayesian inference by enabling exact conditional draws and local moves via user-defined proposal kernels, which directly impact mixing efficiency.
• The method underpins applications in RBMs and quantum algorithms, and hybrid AGS/MH schemes can mitigate high free-energy barriers for faster convergence.

Alternating Gibbs Sampling (AGS) refers to a class of block Gibbs samplers that alternate between updating two (or more) blocks of variables conditionally, and is notably used in latent variable models, model index transdimensional inference, Restricted Boltzmann Machines (RBMs), and quantum algorithms for approximate Gibbs/Boltzmann sampling. AGS is distinguished by its efficiency, exact conditional updates (rather than accept–reject proposals), and convergence properties, with critical applications and limitations determined by the model structure and proposal kernels.

1. General Principles and Formalism

Alternating Gibbs Sampling operates by alternately sampling groups of variables from their full conditionals. In the canonical setting, AGS targets a joint distribution over data $y$, a model index $k$, and parameters $\{\theta^{(j)}\}$, often with $k$ indexing a family of models each with their own parameterization dimensionality. The core AGS update sequence, as formulated in transdimensional contexts (Walker, 2009), consists of:

• Sampling the current model’s parameters $\theta^{(k)}$ from their posterior given the data.
• Proposing neighboring model parameters $\theta^{(k\pm 1)}$ via a user-specified kernel.
• Randomly setting an auxiliary variable $u$ to $k$ or $k+1$ with probability $1-q$ and $q$, respectively.
• Gibbs sampling a new model index $j$ with normalized weights using both forwards and reverse proposal densities, likelihood, and priors.
• Shifting $k \leftarrow j$ and retaining the latent parameter copies.

This Gibbs scheme eliminates Jacobian calculations required in reversible jump MCMC (RJMCMC) and ensures every proposed move is accepted due to exact conditional draws. The same mathematical structure underpins AGS in other domains, such as alternating updates between visible and hidden units in RBMs (Roussel et al., 2021), where $v$ and $h$ are alternately sampled from $P(v|h)$ and $P(h|v)$.

2. AGS in Transdimensional and Model Selection Problems

Walker’s AGS formalism (Walker, 2009) was developed as an alternative to RJMCMC for transdimensional Bayesian inference. Given data $y$, with model index $k$ and model-specific parameters $\theta^{(k)}$, the target posterior is

$$p(k, \theta^{(k)} | y) \propto p(y | \theta^{(k)}, k)\, \pi_k(\theta^{(k)})\, \pi(k)$$

The joint state space includes all latent parameter vectors $\{\theta^{(j)}\}_{j=1}^\infty$, and an auxiliary variable $u$. The AGS alternates between:

1. Sampling $\theta^{(k)}$ from the full conditional posterior.
2. Proposing $\theta^{(k\pm1)}$ via user-defined proposal kernels.
3. Sampling $u$ and then $j$ based on normalized probabilities.
4. Updating the model index and repeating.

Compared to RJMCMC, AGS does not require invertible mappings or Jacobians and has a strictly Gibbs accept-all structure; however, its transitions are typically local in $k$ (i.e., $k \to k\pm1$). Proposals $p(\theta^{(j)}|\theta^{(k)})$ can be chosen arbitrarily but directly affect mixing. The special simplification occurs if proposal kernels and priors satisfy a detailed-balance condition, simplifying the transition weights.

AGS converges to the correct marginal posterior on $(k, \theta^{(k)})$ owing to ergodicity and positivity of the full joint. Slow mixing across models may occur if proposal distributions are poorly chosen or for large jumps across $k$—addressable in principle by redesigning the auxiliary $u$ mechanism (Walker, 2009).

3. Alternating Gibbs Sampling in Restricted Boltzmann Machines

The bipartite structure of RBMs defines an efficient AGS routine for sampling from the model's Boltzmann distribution (Roussel et al., 2021). The RBM joint energy is

$$E(v, h) = -\sum_{i=1}^N \sum_{\mu=1}^M W_{i\mu}\, v_i\, h_\mu + \sum_{i=1}^N V_i(v_i) + \sum_{\mu=1}^M U_\mu(h_\mu)$$

This structure ensures conditional independence within hidden $(h_\mu | v)$ and visible $(v_i | h)$ units given the counterpart, allowing block-Gibbs updates via:

• $h_\mu \sim P(h_\mu | v)$ for all $\mu=1,\dots, M$.
• $v_i \sim P(v_i | h)$ for all $i=1,\dots, N$.

Alternation forms a Markov chain targeting the correct $P(v, h) \propto \exp(-E(v, h))$. The effective marginal energy for $v$ (after integrating out $h$) reads

$$E^{\text{eff}}(v) = \sum_i V_i(v_i) - \sum_\mu \Gamma_\mu(I_\mu(v))$$

where $I_\mu(v) = \sum_i W_{i\mu} v_i$ and $\Gamma_\mu$ denotes the cumulant generating function determined by the hidden unit potential. AGS for RBMs is not, in general, more efficient than local Metropolis–Hastings (MH) sampling on the visible units alone; both are governed by the largest free energy barrier $\Delta f$ between modes, determining an exponential mixing time scaling $\tau \sim \exp(N \Delta f)$.

A key insight is that when the learned hidden-unit representation encodes localized, weakly-overlapping features, augmenting AGS with blockwise MH updates in the hidden space can reduce apparent barriers and accelerate mixing; otherwise, if the representation is highly entangled, no such acceleration is achieved (Roussel et al., 2021).

4. Quantum Alternating Gibbs Sampling: QAOA as Approximate Boltzmann Samplers

AGS also appears in quantum computation in the guise of the Quantum Alternating Operator Ansatz (QAOA) at depth $p=1$, which approximates thermal sampling from classical Hamiltonians (Pelofske, 11 Oct 2025). For an Ising cost Hamiltonian $H_C$ (e.g., the Sherrington–Kirkpatrick model), QAOA alternates a phase-separation unitary $e^{-i\gamma H_C}$ and a mixing unitary $e^{-i\beta H_M}$ starting from the uniform superposition, with two choices for mixer $H_M$:

• X-mixer: $H_M^{(X)} = \sum_i X_i$
• Grover mixer: $$H_M^{(G)} = 2|\!+\!\rangle^{\otimes n} \langle\!+\!|^{\otimes n} - I$$

The output probability distribution

$$P(x; \gamma, \beta) = |\langle x | \psi(\gamma, \beta) \rangle |^2$$

can be interpreted as an approximate Boltzmann law

$$P(x; \gamma, \beta) \approx \frac{e^{-E(x)/T_{\rm eff}}}{Z(T_{\rm eff})}$$

where $T_{\rm eff}(\gamma, \beta)$ is determined by fitting the distribution to minimize a chosen discrepancy (e.g., total variation distance, Kullback–Leibler divergence) against the ideal Boltzmann distribution.

Numerical experiments indicate that, at high effective temperatures $(T_{\rm eff} \gtrsim 10)$ and low total variation distance (TVD $\leq 0.1$), both X- and Grover-mixer QAOA provide good approximations, with the Grover mixer systematically attaining slightly higher $T_{\rm eff}$ at the same error. As the error tolerance is tightened (TVD $\leq 0.01$ or $0.001$), the achievable $T_{\rm eff}$ increases, meaning only near-uniform sampling is possible at such low errors; the Grover mixer outperforms X-mixer in the high-temperature, low-TVD regime due to more uniform treatment of degenerate cost levels (Pelofske, 11 Oct 2025).

5. Mixing Efficiency, Energy Barriers, and Hybrid Schemes

The efficiency of AGS in escaping metastable states is determined by the underlying energy landscape. In high-dimensional models such as mean-field spin glasses or RBMs, both AGS and local MH samplers face the same order $N\Delta f$ energy barriers between free energy minima, resulting in mixing times that scale exponentially with system size. For AGS, the optimal dynamical path between modes can be calculated using large-deviations methods, partitioned into "instanton" (barrier-climbing) and relaxation segments; the action cost corresponds precisely to the free-energy barrier (Roussel et al., 2021).

Hybrid AGS/MH schemes integrate additional Metropolis–Hastings updates in the latent (hidden) variable space after each AGS step. When the hidden-unit representation decomposes as weakly correlated features, MH steps in small blocks encounter much lower energy barriers and can dramatically accelerate mixing. If the hidden structure remains entangled or collective, all-variable updates are needed to match visible space mixing, erasing the speed-up. Empirical demonstrations on Bars-and-Stripes, MNIST, Hopfield, and Lattice Protein datasets reinforce these theoretical insights.

6. Practical Considerations and Limitations

Several caveats and practical notes emerge across AGS variants:

• For transdimensional AGS, the choice of proposal kernel $p(\theta^{(j)}|\theta^{(k)})$ is critical, as poor proposals slow cross-model mixing. Only $k \to k\pm 1$ transitions are supported unless $u$ is redesigned for longer jumps; $q$ tunes up vs. down probabilities; and storage for all $\{\theta^{(j)}\}$ is required, though only two parameters are updated at each iteration (Walker, 2009).
• In RBMs, standard AGS is rarely superior to local MH in terms of mixing across high free energy barriers, except when hidden-unit representations enable localized latent moves.
• For quantum AGS via QAOA, strictly high-temperature (i.e., nearly uniform) Boltzmann sampling is possible at low errors, as improved accuracy compresses the accessible temperature range.
• No Jacobian determinants or invertible transforms are required in any AGS variant, as all transitions are accepted due to the Gibbs property.
• The AGS approach is provably correct for the intended stationary distribution by standard Gibbs sampling theory, provided all joint probabilities are strictly positive.

7. Applications and Empirical Performance

AGS finds application in transdimensional Bayesian inference (e.g., mixture modeling with unknown components), in learning and sampling from RBMs (unsupervised learning, representation extraction), and in quantum-classical thermal sampling. In mixture models, AGS enables efficient component birth/death moves without the reversibility or Jacobian issues of RJMCMC. In RBMs, AGS alternation enables rapid mixing through the bipartite structure, with empirical success on Bars-and-Stripes and Hopfield-type synthetic datasets when the hidden representation structure is appropriate. In QAOA at low depth, AGS enables practical quantum sampling from thermal distributions with quantifiable approximation guarantees at the cost of limited effective temperature control (Walker, 2009, Roussel et al., 2021, Pelofske, 11 Oct 2025).