Diffusion Classifier Overview

• Diffusion Classifier is a generative classifier that repurposes pretrained diffusion models into zero-shot classifiers by using denoising error estimates derived from the ELBO.
• It applies Bayes’ rule to convert a conditional generative model into a discriminative system by comparing expected denoising losses across candidate classes.
• Variants extend the approach to modalities like 3D point clouds and noisy inputs, demonstrating strong compositional reasoning and robustness despite high computational costs.

A Diffusion Classifier (DC) is a generative classifier built from a pretrained diffusion model. In the now-standard usage for image and multimodal recognition, a conditional diffusion model $p_\theta(x_0\mid c)$ is converted into a classifier by Bayes’ rule and by approximating the conditional log-likelihood with the diffusion evidence lower bound (ELBO), so prediction reduces to selecting the label whose conditioning yields the smallest denoising error. The approach is notable because it is fully zero-shot and training-free at inference time: it repurposes a model trained for generation into a discriminative system without additional classifier training (Li et al., 2023, Chen et al., 2024). The phrase “diffusion classifier” also has an earlier graph-learning usage, where labels are propagated over a graph through random-walk diffusions such as Personalized PageRank and the Heat kernel (Berberidis et al., 2018).

1. Definition and scope

In its modern form, DC starts from a conditional generative model and applies Bayes’ rule,

$$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$

With a uniform prior over classes, the prior and evidence terms cancel, leaving

$y^*=\arg\max_c p_\theta(x\mid c).$

Because $p_\theta(x\mid c)$ is intractable for diffusion models, DC replaces exact likelihood evaluation with the diffusion ELBO and uses denoising error as a proxy for conditional log-likelihood (Li et al., 2023).

This construction is conceptually distinct from conventional discriminative zero-shot classifiers such as CLIP-style vision–LLMs. Rather than directly comparing image and text embeddings, DC asks which class condition best explains the observed input under the learned denoising dynamics. That framing is central to claims that diffusion-based recognition exhibits stronger multimodal compositional reasoning than competing discriminative approaches, even when a gap remains on some standard zero-shot benchmarks (Li et al., 2023).

The term is broader than a single architecture. It includes latent text-to-image models such as Stable Diffusion, class-conditional image generators such as DiT, class-conditional 3D diffusion models for point clouds, and extensions that classify Gaussian-corrupted inputs or diffuse directly in label space rather than image space (Li et al., 2023, Koprucu et al., 2024, Chen et al., 2024, Belhasin et al., 25 Nov 2025). This suggests that “DC” is best understood as a decision rule layered on top of diffusion modeling, rather than as a single model family.

2. Probabilistic formulation

The standard diffusion setup defines a forward noising process

$$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$

and trains a network $\epsilon_\theta(x_t,c)$ to predict the added noise. In the $\epsilon$-prediction parameterization, the ELBO reduces, up to a constant, to a negative denoising mean-squared error. Plugging this into Bayes’ rule yields

$$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$

so the predicted class is the one with minimal expected denoising loss (Li et al., 2023).

In practice the expectation is estimated by Monte Carlo. One draws $N$ pairs $\{(t_i,\epsilon_i)\}$ and computes

$$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$0

A practically important detail is to use the same fixed set $$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$1 across all candidate classes, since only relative ELBOs matter and shared samples reduce variance (Li et al., 2023).

Later work generalized this score with explicit timestep weights. In compositional diffusion classification, the loss is written as

$$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$2

and the class posterior is approximated with

$$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$3

This makes timestep weighting a first-class design variable rather than a fixed implementation detail (Jeong et al., 23 May 2025).

A separate theoretical line established that DC is a generative classifier with global smoothness guarantees. For a pretrained diffusion model admitting an ELBO of the form

$$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$4

the resulting posterior inherits an $$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$5 Lipschitz bound, and a Gaussian-corrupted extension yields Noised Diffusion Classifiers (NDCs) suitable for randomized smoothing and certified robustness (Chen et al., 2024).

3. Inference pipeline and implementation practice

For text-to-image DC, each label is mapped to a text prompt, often of the form “a photo of a $$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$6,” and the shared text encoder supplies the conditioning embedding. For class-conditional models such as DiT, the condition is simply the one-hot class index. Zero-shot DC usually assumes a uniform prior, so no explicit calibration is required (Li et al., 2023).

Inference is computationally straightforward but expensive. For every candidate class, the algorithm perturbs the input with sampled noise at one or more timesteps, evaluates the denoiser under that class condition, and accumulates reconstruction errors. The simplest implementation loops over the full class set and all Monte Carlo samples. Because complexity scales roughly as $$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$7, practical work has emphasized variance reduction and pruning. One efficient strategy is an adaptive two-stage scheme that prunes unlikely classes early and allocates more Monte Carlo trials to the leading candidates (Li et al., 2023).

Several implementation choices that are highly consequential for sampling turn out not to help classification. A specific example is classifier-free guidance: $$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$8 Although one can substitute $$y^*=\arg\max_c p(c\mid x), \qquad \log p(c\mid x)=\log p(x\mid c)+\log p(c)-\log p(x).$$9 into the ELBO estimate, the reported ablation found that $y^*=\arg\max_c p_\theta(x\mid c).$0 did not improve zero-shot accuracy. Similarly, cropping the error map to focus on the center yielded no gain, and several importance-sampling schemes for $y^*=\arg\max_c p_\theta(x\mid c).$1 were inferior to standard normal sampling (Li et al., 2023).

The importance of timestep choice has become clearer in later analyses. On ten compositional benchmarks spanning more than 30 tasks, timestep weighting was shown to interact strongly with domain gap, particularly for SD 3-m. Uniform weighting was far from optimal for that model, while low-shot tuning of $y^*=\arg\max_c p_\theta(x\mid c).$2 could move performance from $y^*=\arg\max_c p_\theta(x\mid c).$3 to $y^*=\arg\max_c p_\theta(x\mid c).$4 on CLEVR color binding and from $y^*=\arg\max_c p_\theta(x\mid c).$5 to $y^*=\arg\max_c p_\theta(x\mid c).$6 on WhatsUp-A (Jeong et al., 23 May 2025). A plausible implication is that DC does not merely “read out” a static representation; it depends on which parts of the denoising trajectory are emphasized.

4. Empirical performance and capabilities

The original image-based DC study evaluated zero-shot classification on Food-101, CIFAR-10, Aircraft, Pets, Flowers102, STL-10, ImageNet, and ObjectNet. DC based on Stable Diffusion outperformed a synthetic-data baseline and a supervised ResNet-50 trained on Stable Diffusion features, beat CLIP ResNet-50, and approached OpenCLIP ViT-H/14 on most benchmarks. On compositional reasoning, its Winoground text-score was $y^*=\arg\max_c p_\theta(x\mid c).$7 versus OpenCLIP’s $y^*=\arg\max_c p_\theta(x\mid c).$8, with especially strong performance on relation swaps. For standard ImageNet classification, DC extracted from a DiT-XL/2 model at $y^*=\arg\max_c p_\theta(x\mid c).$9 achieved $p_\theta(x\mid c)$0 top-1, exceeding ResNet-50’s $p_\theta(x\mid c)$1 and ViT-L/32’s $p_\theta(x\mid c)$2 (Li et al., 2023).

Robustness is a recurrent empirical theme. On ImageNet-V2, ImageNet-A, and ObjectNet, diffusion classifiers were reported to lie above the in-distribution versus out-of-distribution linear fit followed by discriminative models, with approximately $p_\theta(x\mid c)$3 effective robustness on ImageNet-A (Li et al., 2023). The certified-robustness line strengthened this picture theoretically and experimentally: APNDC achieved $p_\theta(x\mid c)$4 certified accuracy at $p_\theta(x\mid c)$5 and $p_\theta(x\mid c)$6 at $p_\theta(x\mid c)$7 on CIFAR-10, using a single off-the-shelf diffusion model and no additional data; its clean accuracy was $p_\theta(x\mid c)$8 (Chen et al., 2024).

At the same time, large-scale compositional evaluation showed that the strengths of DC are conditional rather than universal. Across Stable Diffusion 1.5, 2.0, and 3-m, diffusion classifiers often exceeded CLIP on spatial relations, competed on complex attribute binding, underperformed on object recognition, and failed on counting with accuracy below $p_\theta(x\mid c)$9. The same study reported a “generative–discriminative paradox”: SD 3-m was best at generating compositional scenes, but its out-of-the-box DC accuracy on real benchmarks was only approximately $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$0, lower than SD 1.5 and 2.0 at approximately $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$1; yet on its own generated images, SD 3-m’s DC reached approximately $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$2 (Jeong et al., 23 May 2025). This suggests that compositional knowledge and deployable discriminative performance can decouple under domain shift.

5. Variants and extensions

A prominent modality extension is DC3DO, which adapts the diffusion-classifier idea to 3D point clouds. DC3DO uses a class-conditional diffusion model trained on ShapeNet, encodes an input point cloud $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$3 with a PVCNN-based LION encoder into global and local latents, and performs zero-shot classification by summing class-conditioned denoising losses over $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$4 steps. On ShapeNet chairs and cars, evaluated on 200 models per class, DC3DO reached $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$5 on cars and $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$6 on chairs, for an average absolute gain of approximately $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$7 over a multi-view diffusion-classifier baseline; runtime was approximately $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$8 s per object on an NVIDIA T4 GPU (Koprucu et al., 2024).

Another extension modifies the input distribution rather than the modality. Noised Diffusion Classifiers classify Gaussian-corrupted data $$x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\qquad \epsilon\sim\mathcal N(0,I),$$9 by deriving an ELBO for $\epsilon_\theta(x_t,c)$0 and then applying Bayes’ rule to the ELBO approximation. Exact Posterior NDC and Approximated Posterior NDC were introduced as concrete variants, with the latter aligning the posterior approximation with the model’s trained weights and yielding a “free ensemble with no extra cost” (Chen et al., 2024).

A separate line addresses instability due to the choice of matching noise. “Noise Matters” observed that different Gaussian draws can produce substantial performance variation and that existing DCs often ensemble 100–500 noise samples to stabilize predictions. It proposed NoOp, which replaces random noise with a learned dataset-specific base noise plus an image-specific offset generated by a light U-Net. On OxfordPets, combining frequency matching and spatial matching reached approximately $\epsilon_\theta(x_t,c)$1 versus a $\epsilon_\theta(x_t,c)$2 baseline, and on a 4-shot ImageNet source with transfer to eight targets, NoOp improved target average accuracy from $\epsilon_\theta(x_t,c)$3 to $\epsilon_\theta(x_t,c)$4 while reducing overall cost to less than $\epsilon_\theta(x_t,c)$5 of a 5-noise ensemble (Wang et al., 15 Aug 2025).

The label variable itself can also be diffused. Discrete Diffusion Classification Modeling (DiDiCM) treats the class label $\epsilon_\theta(x_t,c)$6 as a one-hot vector and defines a forward diffusion that mixes it toward the uniform distribution, together with a learned reverse process that estimates $\epsilon_\theta(x_t,c)$7. On ImageNet-1k under standard ResNet-50 training, DiDiCM-CP with $\epsilon_\theta(x_t,c)$8 steps achieved $\epsilon_\theta(x_t,c)$9 top-1 and approximately $\epsilon$0 top-5; under corrupted or low-data conditions, reported gains reached $\epsilon$1–$\epsilon$2 absolute top-1 over baseline (Belhasin et al., 25 Nov 2025). Although architecturally different from image-space ELBO ranking, it preserves the central DC idea of classification through reverse diffusion rather than direct label prediction.

6. Limitations, biases, and research directions

The most persistent limitation is computational cost. The original DC framework identified runtime as a major bottleneck because it scales roughly with the number of classes and Monte Carlo samples (Li et al., 2023). In 3D, DC3DO noted that naive inference at $\epsilon$3 would take $\epsilon$4 min per object and that the approach is limited to classes seen during diffusion-model training (Koprucu et al., 2024). In image classification, the noise-instability literature reported that one Pets image can require approximately $\epsilon$5 s on an RTX 3090 for $\epsilon$6 noise samples (Wang et al., 15 Aug 2025). These are not peripheral engineering issues; they shape which variants are practically usable.

Bias analysis has shown that DC’s failure modes differ from those of discriminative vision–LLMs. ASOB-Bench evaluated diffusion classifiers on attribute binding, size-order bias, and background dependency. DC was found to be less prone to attribute misbinding than an OpenCLIP baseline, with lower Concept Association Bias across all seven tested categories. However, on ComCo it was substantially more susceptible to size-order shortcuts, with $\epsilon$7, $\epsilon$8, and a gap of $\epsilon$9, compared with OpenCLIP’s $$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$0. On ImageNet-B it suffered much larger drops under color- and texture-adversarial backgrounds, and even on no-background images it recovered only to $$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$1 versus OpenCLIP’s $$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$2 (Fathi et al., 4 Jul 2026). Reconstruction-error heatmaps and U-Net cross-attention maps localized these biases to specific image regions, indicating that the prompt-conditioned reconstruction score can over-weight large objects and backgrounds.

Domain shift and timestep sensitivity further qualify claims about compositional understanding. The large benchmark study reported that SD 3-m showed the largest cross-domain accuracy drops, up to $$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$3 on Two Objects and $$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$4–$$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$5 on color and spatial tasks, while SD 1.5 and 2.0 lost only approximately $$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$6–$$p_\theta(c_i\mid x)\propto \exp\!\left\{-\mathbb E_{t,\epsilon}\bigl[\|\epsilon-\epsilon_\theta(\sqrt{\bar\alpha_t}x+\sqrt{1-\bar\alpha_t}\epsilon,c_i)\|^2\bigr]\right\},$$7 (Jeong et al., 23 May 2025). This suggests that out-of-the-box DC accuracy may reflect the match between the target visual domain and the model’s generative domain as much as any abstract compositional competence.

The literature’s proposed future directions are correspondingly pragmatic: gradient-based search to avoid looping over all classes, low-resolution proxies, parallel hardware, multimodal conditioning with text plus point cloud, reduced inference steps, improved multi-class scaling, and explicit reweighting or debiasing of timestep and spatial contributions (Li et al., 2023, Koprucu et al., 2024, Fathi et al., 4 Jul 2026). Taken together, these directions indicate that DC has matured from a striking reinterpretation of diffusion likelihoods into a broader research program on generative recognition, one whose central question is no longer whether diffusion models can classify, but under what computational, domain, and robustness conditions they do so reliably.