Interleaved Gibbs Diffusion (IGD)

• Interleaved Gibbs Diffusion (IGD) is a generative sampling framework that interleaves partial Gibbs updates with diffusion model denoising to tackle complex, high-dimensional distributions.
• IGD leverages auxiliary variable splitting and coordinate-wise updates, enabling applications in Bayesian signal separation, blind denoising, and constrained discrete-continuous generation.
• IGD offers rigorous theoretical guarantees and empirical success, demonstrating improved convergence and sample quality in challenging inverse problems and multimodal settings.

Interleaved Gibbs Diffusion (IGD) is a class of generative and posterior sampling algorithms that interleave partial Markov chain Monte Carlo (MCMC)—typically Gibbs sampling—with updates implemented via conditional diffusion models, score-based samplers, or discrete/continuous denoisers. IGD algorithms provide a rigorous mechanism for sampling from complex, high-dimensional distributions—often involving implicit constraints, multimodal structure, or latent Gaussian fields—by alternating tractable conditional draws that leverage the expressive power of modern diffusion models. The framework spans applications in Bayesian signal separation, blind denoising, constrained discrete-continuous generation, inverse problems, and guided sampling under constraints.

1. Foundational Principles and General Framework

The core principle of Interleaved Gibbs Diffusion is to construct an efficient Markov chain over a product space by alternating two (or more) transition kernels, each of which is easy to sample from conditionally but difficult to sample from jointly. In the canonical setup, the full target distribution factorizes as

$\pi(x, z) \propto \pi(x) p(z\,|\,x),$

where $p(z\,|\,x)$ is often a Gaussian noising kernel (continuous $\mathbb{R}^d$) or coordinate-wise masking/mutation (discrete/mixed spaces). Each IGD sweep consists of:

• Diffusion (noising) step: Sample $z \sim p(z\,|\,x)$.
• Denoising (Gibbs) step: Sample $x \sim \pi(x\,|\,z)$, where the conditional is typically intractable, but can be efficiently approximated by a diffusion model, gradient-based MCMC, or learned classifier.

By construction, the Gibbs chain has $\pi(x, z)$ as invariant distribution under regularity conditions (irreducibility, aperiodicity). Extensions involve multiple auxiliary variables, conditional sequential Monte Carlo with resampling (particle Gibbs), or hybridizations with other MCMC kernels. IGD’s design leverages the fact that diffusion-based denoisers or coordinate-wise updates conditionally decompose large intractable sampling problems into tractable low-dimensional subproblems, even when the global posterior is heavily constrained, multimodal, or otherwise inaccessible to gradient-based methods (Anil et al., 19 Feb 2025, Chen et al., 5 Feb 2024).

2. Methodological Variants and Algorithms

Several methodological variants operationalize the IGD paradigm across data domains, forward models, and task types:

a) Bayesian Signal Separation via Plug-and-Play Diffusion-Within-Gibbs.

Given a linear mixture

$$y = \sum_{k=1}^K x^{(k)} + v, \quad v \sim \mathcal{N}(0, \sigma_v^2 I),$$

the posterior $$p(x^{(1:K)} | y) \propto p(y|x^{(1:K)}) \prod_{k=1}^K p_\theta(x^{(k)})$$ is sampled by iterating over $k=1,\ldots,K$. Each $x^{(k)}$ update is equivalent to sampling from the posterior of a denoising problem, which is implemented by a partial reverse-time SDE leveraging a pretrained diffusion model prior for each source. The process is rigorously justified as a valid MCMC sampler converging to the joint posterior in total-variation distance under idealized conditions (Zhang et al., 16 Sep 2025).

b) Mixed Discrete-Continuous Generative Modeling.

For joint spaces $x = (x_d, x_c)$, IGD alternates coordinate-wise noising of discrete and continuous variables according to a deterministic schedule, and reverses each noise injection with an exact coordinate-wise denoising sampler—a classifier for discrete coordinates and a score-based continuous denoiser. The approach generalizes discrete Gibbs chains to hybrid spaces and can enforce implicit constraints without assuming factoriality in the reverse kernel (Anil et al., 19 Feb 2025).

c) Blind Bayesian Denoising.

In settings with unknown or parametrized noise (e.g., $$y = x + \epsilon,\, \epsilon \sim \mathcal{N}(0, \Sigma_\theta)$$), IGD alternates between conditional diffusion-based denoising of $x$ and MCMC updates (e.g., Hamiltonian Monte Carlo) for the noise parameter $\theta$, yielding a fully Bayesian sampler for $(x, \theta\,|\,y)$. The invariant distribution targets the full posterior, and diagnostics such as Gelman–Rubin, ESS, and simulation-based calibration are reported (Heurtel-Depeiges et al., 29 Feb 2024).

d) Diffusive Gibbs Sampling for Multimodal Mixing.

The algorithm introduces an auxiliary “noisy $x$” variable $z$ coupled to $x$ (e.g., $z \sim \mathcal{N}(\alpha x, \sigma^2 I)$). IGD alternates sampling from $$p(x|z) \propto \pi(x)\exp(-\|\alpha x - z\|^2/2\sigma^2)$$ (using MALA/HMC) and $p(z|x)$. The resulting chain overcomes metastability and bridges disconnected modes more rapidly than parallel tempering (Chen et al., 5 Feb 2024).

e) Mixture-based Posterior Sampling in Inverse Problems.

IGD builds a mixture approximation to intermediate posteriors at noise-level $t$ using tractable plug-in functionals. Sampling is realized by augmenting the latent variable space and performing a multi-step Gibbs sweep. Each substep alternates between a “bridge” update for an auxiliary $x_s$, a diffusion denoising from level $s$ to $0$, and noising from $x_s$ to $x_t$. This yields state-of-the-art results for inverse problems in vision and audio (Janati et al., 5 Feb 2025).

The following table summarizes representative IGD algorithms and their domains:

Reference	Domain	Key IGD Features
(Zhang et al., 16 Sep 2025)	Source separation	Plug-and-play priors, interleaved diffusion SDEs
(Anil et al., 19 Feb 2025)	Mixed discrete-continuous gen.	Per-coordinate Gaussian/classifier denoising, constraint handling
(Heurtel-Depeiges et al., 29 Feb 2024)	Blind denoising/cosmology	Alternating diffusion and HMC over parameters
(Chen et al., 5 Feb 2024)	Multimodal sampling (MCMC)	Gaussian auxiliary, MALA/HMC, Metropolis-within-Gibbs
(Janati et al., 5 Feb 2025)	Inverse problems	Mixture posterior, augmented Gibbs with diffusion

3. Theoretical Guarantees and Properties

IGD algorithms are motivated by the invariance of the joint distribution under alternating conditional transitions and benefit from established MCMC theory. Under exact conditional sampling, ergodicity and detailed balance are satisfied w.r.t. the target distribution (Chen et al., 5 Feb 2024). For practical scenarios with approximate conditionals (e.g., learned denoisers), bounds on stationary bias and convergence diagnostics are used to assess error and mixing (Heurtel-Depeiges et al., 29 Feb 2024).

For plug-and-play diffusion within Gibbs (e.g., source separation), the procedure provably converges in total-variation to the true posterior as the number of outer Gibbs rounds increases and the inner diffusion samples are ideal (infinitesimal stepsize, perfect score estimate) (Zhang et al., 16 Sep 2025). Similarly, in mixture-based IGD, theoretical analysis justifies that sequentially composed Gibbs kernels target the desired mixture distribution, with KL bounds directly related to the accuracy of plug-in potentials (Janati et al., 5 Feb 2025).

4. Empirical Performance and Applications

IGD demonstrates strong empirical performance across a range of domains:

• Signal separation: Heartbeat extraction from noisy mixtures, outperforming retrained monolithic diffusion models and methods that lack modular plug-and-play priors (Zhang et al., 16 Sep 2025).
• Constraint generation: 3-SAT instance generation and molecule generation on QM9, where IGD captures non-factorizable dependencies, yielding higher clause satisfaction and chemical validity than factorized diffusion samplers (Anil et al., 19 Feb 2025).
• Blind denoising: Posterior mean and uncertainty quantification for images and CMB maps within fully Bayesian models, achieving higher PSNR/SSIM than BM3D and DnCNN (Heurtel-Depeiges et al., 29 Feb 2024).
• Inverse problems: Superresolution, inpainting, and audio source separation—all with minimal or no retraining. On FFHQ/ImageNet, IGD attains first or second place in LPIPS metrics across various tasks (Janati et al., 5 Feb 2025).
• SAR imaging: Split-Gibbs IGD achieves $>$7 dB PSNR gain in simulations, with improved sidelobe suppression and preservation of scene details on real Sentinel-1A data (Gao et al., 2 Dec 2025).
• Multimodal MCMC: Faster mode mixing and lower estimation error than parallel tempering in high-dimensional Bayesian neural networks and molecular conformation estimation (Chen et al., 5 Feb 2024).

Ablation studies consistently confirm the monotonic gains in sample quality as inference-time Gibbs rounds increase, and demonstrate robustness across priors, noise levels, and architecture choices.

5. Design Choices and Extensions

Key aspects of IGD design include:

• Choice of auxiliary variables: Gaussian mixtures, bridges, or masking are standard.
• Conditional updates: Implemented via pretrained denoisers (score networks or classifiers), Langevin/Metropolis-within-Gibbs, or SMC in discrete settings.
• Interleaving patterns: Coordinate-wise, round-robin, block-wise, or hybrid patterns optimize between sequential dependency capture and parallelization (Anil et al., 19 Feb 2025).
• Refinement and conditioning: IGD allows post-hoc sample refinement (“ReDeNoise”), flexible inclusion of hard/soft constraints via state-space doubling and masking, and accommodates multi-channel or convolutive models in applications such as source separation and remote sensing (Zhang et al., 16 Sep 2025, Gao et al., 2 Dec 2025).
• Mixture and split Gibbs: IGD generalizes to handle mixture approximations or auxiliary splitting, as in SAR and pixel/latent posterior inference (Janati et al., 5 Feb 2025, Gao et al., 2 Dec 2025).

Scheduled coupling parameters, noise schedules, and number of inner/outer rounds are tuned empirically for stability and performance.

6. Theoretical and Practical Limitations

While IGD offers strong theoretical guarantees in the ideal setting, practical limitations arise from:

• Approximate denoisers: Learned score functions can introduce stationary bias; simulation-based calibration and effective sample size are used for diagnostics (Heurtel-Depeiges et al., 29 Feb 2024).
• Computational cost: Each Gibbs iteration typically requires reverse diffusion or multiple gradient steps; scalable variants are explored (e.g., efficient DDIM, SMC-based sampling) (Dang et al., 11 Jul 2025).
• Dependency structure: Sequential updates capture dependencies but can be slower than fully factorized samplers, particularly in high-dimensional settings. Parallelization and multi-block updates are potential remedies (Anil et al., 19 Feb 2025).
• Domain restrictions: Assumptions of additive Gaussian noise, or of the tractability of per-step conditionals, may limit application in some non-Gaussian or intricately constrained problems (Heurtel-Depeiges et al., 29 Feb 2024, Anil et al., 19 Feb 2025).

7. Current Impact and Research Directions

IGD has influenced domains including Bayesian signal processing, discrete and mixed generative modeling, statistical inverse problems, and high-dimensional MCMC. It enables modular, “plug-and-play” combination of pretrained generative priors and offers a principled trade-off between accuracy and compute through inference-time refinement. Empirical evidence shows that investing more sampling effort (Gibbs steps) reliably improves sample quality and posterior fidelity without retraining, a key advantage in both research and applied pipelines.

Open directions for IGD include flow-matching/ODE-based updates, non-Gaussian likelihood extensions, hybrid parallel samplers, and further reductions of computational cost via approximate but unbiased conditional kernels (Anil et al., 19 Feb 2025).

---

References:

• "Bayesian Signal Separation via Plug-and-Play Diffusion-Within-Gibbs Sampling" (Zhang et al., 16 Sep 2025)
• "Interleaved Gibbs Diffusion: Generating Discrete-Continuous Data with Implicit Constraints" (Anil et al., 19 Feb 2025)
• "Listening to the Noise: Blind Denoising with Gibbs Diffusion" (Heurtel-Depeiges et al., 29 Feb 2024)
• "Diffusive Gibbs Sampling" (Chen et al., 5 Feb 2024)
• "A Mixture-Based Framework for Guiding Diffusion Models" (Janati et al., 5 Feb 2025)
• "Diffusion-Prior Split Gibbs Sampling for Synthetic Aperture Radar Imaging under Incomplete Measurements" (Gao et al., 2 Dec 2025)
• "Inference-Time Scaling of Diffusion LLMs with Particle Gibbs Sampling" (Dang et al., 11 Jul 2025)