Classifier-Guided Diffusion

• Classifier-guided diffusion is a conditional generation paradigm that steers the reverse process of diffusion models using gradient signals from an auxiliary classifier.
• It encompasses both direct classifier guidance and classifier-free methods, balancing fidelity and diversity through calibrated score modifications.
• Recent extensions enhance gradient stability and efficiency, broadening applicability to image synthesis, speech, medical imaging, and multi-objective optimizations.

Classifier-guided diffusion is a conditional generation paradigm wherein a diffusion model’s sampling trajectory is steered using gradient signals from an auxiliary classifier. Originally developed for conditional image and speech synthesis, the framework has found broad applicability due to its flexibility, ability to decouple generative and conditional components, and empirical performance. Recent advances expand the methodology to robustify its gradients, eliminate the need for explicit classifiers (“classifier-free” guidance), support diverse objectives (e.g., fairness, preference dominance, adversarial robustness), and enable guidance with non-robust or even gradient-free classifiers.

1. Foundations of Classifier-Guided Diffusion

Classifier guidance leverages a trained diffusion probabilistic model (DPM)—an iterative generative model that progressively denoises a Gaussian-noise sample—by modifying the reverse-time process to incorporate conditioning information. In the basic form (for a denoising diffusion probabilistic model, DDPM), the reverse SDE for the latent $x_t$ is:

$$dx_t = [f(x_t, t) - g^2(t)\nabla_{x_t} \log p_t(x_t)]\,dt + g(t)\,d\bar{w}$$

To condition upon auxiliary information $y$ (such as a class label or transcript), classifier guidance augments the score term:

$$\nabla_{x_t} \log p(x_t|y) = \nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p(y|x_t)$$

Thus, during each denoising step, a classifier $p_\phi(y|x_t)$ predicts the probability of $y$ given the current latent, and the gradient $\nabla_{x_t} \log p_\phi(y|x_t)$ is added—scaled by a guidance strength parameter $s$ and, optionally, a normalization or calibration term.

This mechanism was first formalized for text-to-speech in "Guided-TTS" (Kim et al., 2021), for image synthesis in DDPMs and subsequent generative architectures (Ho et al., 2022, Ma et al., 2023), and adapted across modalities and downstream tasks.

2. Guidance Mechanisms and Generalizations

2.1. Direct Classifier Guidance

Traditional classifier guidance requires a classifier trained on noisy latents (i.e., robust to the noise schedule of the diffusion process) and applies its gradient at each denoising step (Ho et al., 2022). The update equation in discretized DDPMs is:

$$\mu_t' = \mu_t + s\cdot \Sigma_t(x_t) \cdot \nabla_{x_t} \log P(y|x_t)$$

where $\mu_t$ is the predicted mean of the denoising distribution at $t$, and $\Sigma_t$ is its covariance.

2.2. Classifier-Free Guidance

To avoid training separate robust classifiers, "classifier-free guidance" jointly trains a diffusion model to output both conditional and unconditional scores (Ho et al., 2022). At inference, the guided score is computed as:

$$\tilde{s}_\theta(x, y) = (1 + w) s_\theta(x, y) - w s_\theta(x)$$

where $w$ trades off mode coverage versus sample fidelity. This approach has been widely adopted, as in Guided-TTS 2 (Kim et al., 2022), OSCAR (Zaland et al., 12 Feb 2025), and SLCD (Oertell et al., 27 May 2025), among others.

2.3. Robust, Gradient-Free, and Non-Robust Classifier Guidance

High-quality classifier guidance depends critically on stable and meaningful gradients. Robust classifiers, adversarially trained on noise-matched latents, yield gradients that are both perceptually aligned and stable, significantly improving sample quality (Kawar et al., 2022). However, non-robust classifiers exhibit unstable, noisy gradients and may harm conditional synthesis. Recent work, including (Vaeth et al., 1 Jul 2025, Vaeth et al., 25 Jun 2024), demonstrates that stabilization techniques—such as using a one-step denoised estimate for the classifier input and applying moving average or ADAM-like smoothing—significantly improve gradient stability and sample quality even for non-robust classifiers.

Gradient-free methods like GFCG (Shenoy et al., 23 Nov 2024) replace gradient computations with confidence-based forward inference, adaptively modulating guidance strength via the classifier's output probabilities.

2.4. Calibration, Pre-Conditioning, and Design Choices

Calibration and pre-conditioning techniques rescale classifier logits, normalize gradients, or introduce adaptive temperature parameters to better match classifier gradients to the diffusion model's dynamics (Ma et al., 2023). Ablation studies reveal that careful calibration and normalization can substantially improve the utility of off-the-shelf classifiers in guiding high-quality conditional synthesis.

3. Algorithmic Implementations and Technical Details

3.1. Formalized Sampling Procedures

The central innovation lies in modifying the step-wise transition dynamics of the diffusion model. For example, in Guided-TTS (Kim et al., 2021), the update is:

$$X_{t - \Delta t} = X_t + \frac{\beta_t}{N} \left[ \frac{1}{2}X_t + \nabla_{X_t} \log p_\theta(X_t) + s\,\alpha_t \nabla_{X_t} \log p_\phi(\hat{y} | X_t) \right] + \sqrt{\beta_t/N} z_t$$

where $s$ is the gradient scale, $\alpha_t$ is norm-based scaling (equalizing the norm of the unconditional and classifier gradients per Equation 7), and $z_t$ is Gaussian noise.

For classifier-free guidance, the update is typically:

$$\hat{\epsilon}_t = (1 + s)\,\epsilon_\theta(x_t, t, y) - s\,\epsilon_\theta(x_t, t, \emptyset)$$

Offline and online feature regularization and clustering via optimal transport (Sinkhorn-Knopp) are used for self-guidance (Hu et al., 2023).

3.2. Robustness and Stabilization Procedures

To address gradient instability, especially for non-robust classifiers, stabilization procedures such as exponential moving averages (EMA) and ADAM-style normalization are used:

$$\nu_t^{\text{ema}} = \beta \nu_{t-1}^{\text{ema}} + (1 - \beta)g_t$$

where $g_t$ is the classifier guidance gradient at step $t$; in ADAM-style, normalization accounts for running variance.

3.3. Guidance for Multi-Objective and Complex Tasks

Composite objectives can be handled by summing or scaling multiple classifier gradients, e.g., for fairness-aware generation (Lin et al., 13 Jun 2024), one term steers toward target labels ($\nabla_x \log p(y|x)$), another maximizes sensitive attribute entropy for fairness ($\nabla_x H(p_{\theta_z}(z|x))$). For preference-guided optimization (Annadani et al., 21 Mar 2025), a preference classifier's gradient ($\nabla_x \log p_\phi(x, r|t)$) encodes dominance relations in multi-objective design.

4. Practical Applications and Empirical Findings

Classifier-guided diffusion has been successfully applied to:

• High-fidelity image synthesis (Ho et al., 2022, Kawar et al., 2022, Ma et al., 2023) (FID $<3.0$, with improved precision/diversity trade-off).
• Text-to-speech without transcripts (Kim et al., 2021, Kim et al., 2022) (MOS $\sim$4.2, CER $<1.1\%$ in LJSpeech).
• Adversarial purification (Zhang et al., 12 Aug 2024), where classifier confidence guidance enhances robustness under strong attacks (AutoAttack, BPDA), e.g., robust accuracy $>73\%$ on CIFAR-10/l∞ threat.
• Medical imaging: conditional diffusion with a discriminative embedding-based classifier yields accuracy $83\%$ and F1 $0.858$ in diagnosing diabetic foot ulcer infection (Busaranuvong et al., 1 May 2024).
• Federated learning: classifier-free guidance achieves $>$99% reduction in client-server communication overhead while outperforming classifier-guided baselines (Zaland et al., 12 Feb 2025).
• Multi-objective optimization: classifier-guided diffusion efficiently discovers diverse Pareto-optimal solutions, outperforming prior inverse/generative approaches (Annadani et al., 21 Mar 2025).
• Controlled content generation: SLCD steers generation toward high-reward regions under a KL constraint, with theoretical no-regret online learning convergence (Oertell et al., 27 May 2025).
• Semantic editing and disentanglement: classifier-guided embedding optimization enables precise, prompt-free edits in text-to-image diffusion models (Chang et al., 20 May 2025).

5. Limitations, Trade-Offs, and Design Considerations

5.1. Gradient Quality and Stability

Successful guidance demands stable and meaningful classifier gradients. Robust or adversarially trained classifiers yield gradients that correlate with human perception; non-robust classifiers typically fail unless stabilized (Kawar et al., 2022, Vaeth et al., 1 Jul 2025, Vaeth et al., 25 Jun 2024).

5.2. Cost and Scalability

Gradient computations through classifiers are computationally expensive, especially in large models or multi-modal tasks. Classifier-free and gradient-free methods offer improved efficiency (Ho et al., 2022, Shenoy et al., 23 Nov 2024), enabling scaling to high resolutions and federated environments (Zaland et al., 12 Feb 2025).

5.3. Sample Quality vs. Diversity

Guidance strength parameters ($s$, $w$, $\lambda$) trade off precision/fidelity against diversity. Overly high guidance sharpens samples at the cost of mode coverage, potentially collapsing diversity (Ho et al., 2022, Ma et al., 2023).

5.4. Generalization and Adaptation

Off-the-shelf classifiers with careful pre-conditioning and norm-based scaling can be effective (Ma et al., 2023). Classifier-based guidance is extendable to new speakers, tasks, or conditions without retraining the base diffusion model (as in zero-shot voice adaptation (Kim et al., 2022)).

6. Extensions and Research Directions

Recent work explores:

• Gradient-free guidance, allowing for highly efficient conditional sampling (Shenoy et al., 23 Nov 2024).
• Unified frameworks for fairness, adversarial robustness, and preference- or reward-guided generation (Lin et al., 13 Jun 2024, Zhang et al., 12 Aug 2024, Annadani et al., 21 Mar 2025, Oertell et al., 27 May 2025).
• Robust classifier guidance for domains beyond images, including speech, medical imaging, 3D generation, and biological sequences (Kim et al., 2021, Busaranuvong et al., 1 May 2024, Zhang et al., 2023, Oertell et al., 27 May 2025).
• Self-supervised or self-guided diffusion, dispensing with explicit classifiers by extracting pseudo-labels from discriminative features within the diffusion model (Hu et al., 2023).

A key trend is the generalization of guidance to arbitrary objectives, including non-differentiable constraints, latent edit controls, and structural optimization.

7. Summary Table: Guidance Approaches

Guidance Method	Classifier Training	Gradient Use	Robustness Requirement	Scalability / Cost
Classic Classifier Guidance	Noisy data required	Backprop	Yes	High (all steps)
Classifier-Free Guidance	N/A	None (score diff)	No	Low
Robust Classifier Guidance	Adversarial/noisy	Backprop	Yes (per SDE/noise)	High
Gradient-Free Guidance	Any	None (inference)	No	Very Low
Self-Guidance	N/A (self-imposed)	None	N/A	Low

Classifier-guided diffusion thus encompasses a wide variety of algorithmic and practical techniques for conditional generation with diffusion models. This paradigm enables highly flexible, scalable generative modeling—adaptable to diverse domains and objectives, contingent fundamentally on the stability, fidelity, and calibration of classifier-derived guidance signals.