Consistent Annealed Sampling (CAS)

• Consistent Annealed Sampling is a method for iterative sampling that rigorously enforces an annealing schedule to mitigate noise drift and mode collapse in high-dimensional distributions.
• It systematically combines score-driven updates with scheduled noise injection, blending unconditional and conditional generative approaches to preserve variance and improve convergence.
• CAS provides provable guarantees and enhanced empirical performance in diffusion models, posterior sampling, and particle methods, leading to superior mode coverage and sample fidelity.

Consistent Annealed Sampling (CAS) is a rigorous framework for iterative sampling from high-dimensional distributions, particularly in the context of score-based generative models, posterior inference with diffusion models, and kernel particle methods. The central objective of CAS is to enforce consistency with a prescribed “annealing” (noise or temperature) schedule during discretized sampling, thereby addressing issues that arise from drift in marginal noise (in Langevin/diffusion methods) or poor mode coverage (in particle optimization methods) when using imperfect or limited samplers. CAS achieves provable guarantees and improved empirical performance by systematically combining score-driven updates with carefully tuned injection of noise, blending unconditional and conditional generative frameworks, and, in some cases, kernelized repulsion.

1. Formal Definition and Mathematical Foundations

CAS aims to draw samples $x$ from either a target distribution $p(x)$ or a conditional posterior $p(x\mid y)$, typically when only a learned approximation of the score $\nabla_x \log p(x)$ is available and computational constraints limit the number of iterative steps or the accuracy of the approximation.

In score-based generative models, CAS is used to sample from smoothed versions $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$, descending a schedule of noise levels $\{\sigma_i\}$ geometrically from a large $\sigma_1$ (covering the data support) to a small $\sigma_N$ (close to the data manifold). The CAS update at each step is:

$$x_i = x_{i-1} + \eta \, \sigma_i^2 s_\theta(x_{i-1}, \sigma_i) + \beta\, \sigma_{i+1} z_i$$

where $z_i\sim\mathcal N(0,I)$, $p(x)$0 approximates the score, and $p(x)$1 are set so that $p(x)$2 is preserved. This ensures that with finite steps and possibly imperfect $p(x)$3, the sample sequence adheres exactly to the intended marginal schedule, a property not maintained by standard Annealed Langevin Sampling (ALS) when $p(x)$4 is moderate or $p(x)$5 inexact (Serrà et al., 2021).

In posterior sampling, CAS combines unconditional diffusion samplers (to initialize near $p(x)$6) and an annealed chain of Langevin refinements targeting posteriors $p(x)$7 for a descending noise schedule $p(x)$8. The algorithm avoids error amplification by ensuring the chain never deviates far from regions of reliable score estimation, and carefully controls the accumulation of estimator and discretization errors (Xun et al., 30 Oct 2025).

For particle approximations, CAS is realized as annealed Stein Variational Gradient Descent (annealed SVGD), with a temperature schedule $p(x)$9, and incorporates temperature-weighted gradients for incremental exploration-to-exploitation transitions (d'Angelo et al., 2021).

2. Noise and Temperature Scheduling

CAS strictly enforces a geometric noise/temperature schedule:

$p(x\mid y)$0

and parameterizes the update weights as:

$p(x\mid y)$1

with $p(x\mid y)$2. This reparameterization guarantees that, irrespective of the finite budget $p(x\mid y)$3, the updates remain within stability/variance-preservation bounds: $p(x\mid y)$4 and $p(x\mid y)$5 (Serrà et al., 2021). For temperature-based schemes, inverse-temperature schedules $p(x\mid y)$6 (linear, tanh, or cyclical) are used to generate tempered targets $p(x\mid y)$7, allowing annealed SVGD to interpolate between exploratory (low $p(x\mid y)$8) and exploitative (high $p(x\mid y)$9) regimes (d'Angelo et al., 2021).

3. Algorithmic Structure and Theoretical Guarantees

CAS algorithms are characterized by alternating steps of drift (using the score approximation) and controlled stochasticity (noise injection), with explicit correction for the discretization-induced deviation from the intended schedule. In diffusion/posterior applications (Xun et al., 30 Oct 2025), the procedure is:

• Draw initial $\nabla_x \log p(x)$0 using unconditional diffusion.
• For $\nabla_x \log p(x)$1 to $\nabla_x \log p(x)$2:
  • Define $\nabla_x \log p(x)$3.
  • Use updated score $\nabla_x \log p(x)$4.
  • Apply a short burst of Langevin steps of duration $\nabla_x \log p(x)$5 and step size $\nabla_x \log p(x)$6.
• Return $\nabla_x \log p(x)$7 as an approximate sample from $\nabla_x \log p(x)$8.

Under assumptions of $\nabla_x \log p(x)$9-strong log-concavity, $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$0-Lipschitzness, and an $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$1 error bound on the score approximation, CAS guarantees polynomial mixing time and samples with bounded total variation distance to the true posterior. The required error bound is $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$2 in the score estimator (as opposed to sub-exponential error required for vanilla Langevin), and the total oracle complexity is polynomial in $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$3, where $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$4 (Xun et al., 30 Oct 2025).

For denoising score matching (Serrà et al., 2021), final-step "Expected Denoised Sample" (EDS) correction is used:

$p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$5

For particle samplers, the annealed SVGD update at step $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$6 is:

$p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$7

guaranteeing weak convergence to the target density $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$8 as $p_\sigma(x) = (p*\mathcal N(0,\sigma^2 I))(x)$9 and under appropriate step-size decay (d'Angelo et al., 2021).

4. Connections to Related Sampling Schemes

CAS generalizes and interpolates between several existing samplers:

Scheme	Special CAS Parameterization	Key Difference
Annealed Langevin Sampling	$\{\sigma_i\}$0 ($\{\sigma_i\}$1)	Noise amplitude scaled differently
Predictor–Corrector SDE	$\{\sigma_i\}$2	PC predictor up to factor $\{\sigma_i\}$3
Deterministic Denoising	$\{\sigma_i\}$4 ($\{\sigma_i\}$5)	Pure denoising, no noise injection
Full Noise Injection	$\{\sigma_i\}$6 ($\{\sigma_i\}$7)	Pure “denoise then re-noise”

In the limit $\{\sigma_i\}$8 ($\{\sigma_i\}$9), CAS converges to ALS or PC predictor schemes (Serrà et al., 2021).

For annealed SVGD, CAS is compared to:

• Standard SVGD (deterministic updates, prone to mode collapse on multimodal targets).
• Noisy/stochastic SVGD, simulated tempering, and cyclical annealing in stochastic gradient Langevin dynamics (SGLD), which may introduce additional randomness or require parallel chains (d'Angelo et al., 2021).

5. Implementation and Practical Tuning

Practical application of CAS requires setting endpoints for the noise schedule $\sigma_1$0 or temperature schedule $\sigma_1$1, total step count $\sigma_1$2, and the schedule parameter $\sigma_1$3. Empirically, $\sigma_1$4 is effective across a wide $\sigma_1$5 range ($\sigma_1$6). Tuning can be performed by sweeping $\sigma_1$7 logarithmically and selecting values via perceptual or likelihood metrics (Serrà et al., 2021).

For posterior sampling in inverse problems (Xun et al., 30 Oct 2025), CAS alternates diffusion-based warm starts with short, annealed conditional Langevin refinements, and step sizes are chosen so that overall discretization error remains controlled ($\sigma_1$8). Warm starts can use any unconditional sampler providing $\sigma_1$9-accurate scores (e.g., DDIM).

6. Empirical Results and Applications

In high-dimensional inverse problems such as image inpainting, super-resolution, and deblurring, CAS was empirically validated on the FFHQ-256 dataset. Starting from a diffusion-predicted sample, CAS Langevin refinement further reduced per-image $\sigma_N$0 error below that of baseline diffusion-posterior sampling (DPS), while the Fréchet Inception Distance (FID) remained comparable or improved for small step sizes. CAS was observed to better preserve fine detail and structure in inpainting and super-resolution tasks (Xun et al., 30 Oct 2025).

In annealed SVGD, CAS empirically achieves superior mode coverage and weight reconstruction compared to standard SVGD, particularly on highly multimodal distributions and in high-dimensional settings. On various benchmarks, mode coverage improved from 1 to the full number of modes, and mean maximum discrepancy (MMD) was reduced by up to 80% (d'Angelo et al., 2021).

7. Significance and Theoretical Implications

Consistent Annealed Sampling unifies a broad class of iterative sampling schemes by addressing failure modes associated with finite-step schedules and imperfect score approximations. In diffusion/posterior contexts, it is the first method to provide polynomial-time guarantees for posterior sampling under an $\sigma_N$1 score error, rather than requiring strong exponential error bounds. In particle-based samplers, it enables deterministic, single-chain exploration of complex multimodal targets, preserving all favorable convergence properties of classical SVGD.

A plausible implication is that many existing denoising-diffusion and score-based sampling schemes can be reformulated as special instances or limits of CAS, which encourages uniform adoption of consistent scheduling as a standard for robust generative modeling and posterior inference.

---

Selected References

• Posterior Sampling by Combining Diffusion Models with Annealed Langevin Dynamics (Xun et al., 30 Oct 2025)
• On tuning consistent annealed sampling for denoising score matching (Serrà et al., 2021)
• Annealed Stein Variational Gradient Descent (d'Angelo et al., 2021)