How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation

Abstract: Diffusion models have recently been repurposed for zero-shot classification, giving rise to diffusion classifiers that identify the best-matching text prompt by minimizing the noise-prediction error. Despite their growing adoption, how these models make classification decisions remains poorly understood. We introduce ASOB-Bench, a bias evaluation for diffusion classifiers along three dimensions: Attribute binding, Size-Order bias, and Background dependency. These dimensions serve not as an exhaustive taxonomy but as targeted probes of how the text-conditioned reconstruction-error score reaches a decision. Such a perspective is well studied for discriminative vision-LLMs, yet remains overlooked for diffusion classifiers. Extending an existing framework with five new attribute categories on newly constructed datasets, we find diffusion classifiers are less prone to attribute misbinding than an OpenCLIP baseline; on the established ComCo benchmark they are substantially more susceptible to size-order shortcuts; and on ImageNet-B they suffer far larger accuracy drops, revealing heavy reliance on background over foreground cues. Reconstruction-error heatmaps and U-Net cross-attention visualizations expose the mechanism behind each bias. Because diffusion classifiers share the same denoiser as text-to-image models, these single-pass diagnostics also point toward analogous failure modes in generation. Overall, diffusion classifiers exhibit a distinct bias profile from vision-LLMs, offering guidance for building more robust diffusion-based models.

• The paper introduces ASOB-Bench to systematically evaluate diffusion classifiers by analyzing attribute binding, size-order bias, and background dependency.
• It demonstrates that, while diffusion classifiers reduce attribute misbinding on natural images, they are vulnerable to spatial and contextual shortcuts in compositional tasks.
• The results highlight the need for refined error aggregation and architectural adjustments to mitigate inherent reconstruction-error based biases.

Decision Mechanisms and Biases in Diffusion Classifiers: Analysis via ASOB-Bench

Introduction

Diffusion models have established themselves as the de facto standard for generative modeling in vision, enabling high-fidelity text-to-image synthesis via iterative denoising. A recent paradigm shift is the repurposing of these generative models for zero-shot classification: "diffusion classifiers" predict class labels by selecting the text prompt whose conditioning minimizes the model’s denoising error on a given image. Despite competitive performance relative to contrastive models (e.g., CLIP), the precise inductive biases and decision mechanisms of diffusion classifiers are poorly understood.

This paper introduces ASOB-Bench, a framework for systematic bias analysis in diffusion classifiers across three targeted axes: attribute binding, size-order bias, and background dependency. Through comprehensive controlled experiments, mechanistic visualizations, and comparison to an OpenCLIP baseline (sharing the same text encoder), the study rigorously dissects failure modes inherent to the diffusion classification paradigm. The findings delineate a distinct bias profile, highlighting critical vulnerabilities of reconstruction-error-based decision rules and offering guidance for future development.

Figure 1: Overview of ASOB-Bench across three bias dimensions: attribute binding, size-and-order, and background dependency. Reconstruction-error heatmaps and cross-attention visualizations provide mechanistic insight into each bias.

Methodology and Evaluation Design

The classification rule in diffusion classifiers is formulated as prompt selection that minimizes noise-prediction error over sampled denoising timesteps. The analysis covers the following bias dimensions:

• Attribute binding: The degree to which models misattribute attributes (e.g., color, shape) to incorrect objects within a multi-object scene—a structural weakness often formalized as Concept Association Bias (CAB).
• Size-order bias: The systematic over-reliance on spatial salience or prompt order, leading to shortcut reasoning, especially in compositional scenes where object area or mention order is confounded.
• Background dependency: The extent to which global or spurious background cues unduly influence prediction, making the model vulnerable to contextual or adversarial perturbations.

Tailored datasets were composed for each axis, spanning both natural and synthetically generated attribute combinations, the ComCo compositionality benchmark, and ImageNet-B with controlled backgrounds. Mechanistic insight was gained by leveraging denoising reconstruction-error heatmaps and U-Net cross-attention visualizations.

Attribute Binding Bias: Quantitative and Mechanistic Dissection

Diffusion classifiers demonstrate markedly lower attribute-misbinding rates than OpenCLIP. Across seven attribute categories (including color, part-whole, material, shape, size, and temperature), CAB metrics and two-object accuracies consistently favor diffusion classifiers. Notably, in the classic color binding task on natural images, diffusion classifiers yield a two-object accuracy of 0.212 vs. OpenCLIP’s 0.112; CAB scores also trend lower (0.714 vs. 0.8775).

However, this advantage is contingent on the presence of prototypical attribute–object co-occurrences. Synthetic datasets with "unnatural" attribute combinations abolish the misbinding gap, supporting a mechanism rooted in distributional priors rather than architectural invariance.

Figure 2: One-step reconstruction error heatmaps for natural (top) and unnatural (bottom) color pairs. On natural images, error concentrates on the distractor eggplant. On unnatural images, error correctly shifts to the queried carrot, confirming misbinding stems from learned color--object co-occurrences.

Figure 3: Cross-attention heatmaps for ``carrot'' under natural (top) and unnatural (bottom) conditions. Attention is routed by color cue regardless of prototypical associations, confirming this is structural to the cross-attention mechanism.

Figure 4: Full reconstruction for a table-and-clock scene. Square-table prompt causes square forms to leak onto the round clock, confirming that attributes physically migrate onto the distractor during prediction.

Empirical visualization confirms these quantitative trends:

• Reconstruction error heatmaps localize model error to the distractor region in CAB-failure cases, but correctly shift to the queried object in unconfounded scenarios.
• Cross-attention maps demonstrate structurally color-driven routing, invariant to whether co-occurence priors are present, highlighting that attention assignment is dictated by surface cues provided by prompt conditioning.

Temporal analysis reveals bias accentuation at higher diffusion timesteps ($t \geq 600$), coinciding with the regime recommended for improved attribute recognition, thus suggesting an inherent trade-off between accuracy and misbinding.

Size-Order Bias: Shortcut Vulnerability in Compositional Tasks

In size-order controlled evaluations (ComCo), accuracy in bias-aligned scenarios (largest object changed, listed first) remains high for both architectures. Diffusion classifiers, however, exhibit a large accuracy drop in bias-conflicting settings (largest object unchanged, smaller object altered): accuracy collapses to 0.585 vs. 0.953 in the aligned setting (gap: 0.37 for diffusion vs. 0.23 for CLIP).

Figure 5: Error heatmaps for negative prompts of Scenario 1 (a) and Scenario 2 (b). Error correctly concentrates on the changed object, yet the fork's large spatial footprint generates sufficient residual error to make the negative prompt competitive.

Mechanistically, prompt selection aggregates reconstruction error over all pixels, causing large (even irrelevant) objects to dominate the error budget, and thus obscure semantically salient changes. Temporal stratification confirms that this area-weighted shortcut is not primarily dictated by token order, but by spatial extent within the image.

Background Dependency: Structural Entanglement With Context

Diffusion classifiers show acute sensitivity to background manipulations. On ImageNet-B, adversarial color or texture alterations decrease accuracy by 15–19 percentage points (e.g., 91.4% to 72.4%), whereas OpenCLIP experiences only ~2 point reduction. The presence of the background is crucial; removing context completely fails to recover maximal accuracy for diffusion, again underscoring the entanglement:

Figure 6: Reconstruction-error heatmaps across background conditions. Even in non-adversarial settings, error concentrates on the background. Under adversarial perturbation (color, texture), this background focus intensifies.

Temporal error analysis further reveals that across all timesteps, error disproportionately accumulates in background regions, especially when those regions are visually salient or adversarially modified, supporting the conclusion that classification in these models is not foreground-centric.

Architectural Generalization

Replication on Stable Diffusion 3 (DiT architecture, flow-matching objective) qualitatively recapitulates all bias dimensions, albeit with absolute lower accuracy due to training data and scale differences. This indicates that the observed biases arise from the general decision rule and not from a specific backbone architecture.

Implications and Future Directions

These findings have significant practical and theoretical implications for the use and further development of diffusion classifiers:

• Robustness in compositional reasoning: While less prone to attribute-misbinding, diffusion classifiers remain susceptible to global context and salience-based shortcuts due to the aggregate nature of the reconstruction-error scoring rule.
• Generation failure modes: Since diffusion classifiers and standard text-to-image models share underlying architectures, the same structural biases are expected to manifest in generative tasks, especially where compositionality and context matter.
• Model selection and deployment: Headline accuracy is insufficient as a reliability metric for deployment in the wild; bias-centric diagnostics (as enabled by ASOB-Bench) are essential for anticipating and mitigating failure.
• Mitigation strategies: Adjustments to error aggregation (e.g., region-aware weighting), augmenting prompt design, or explicit disentanglement of background and foreground during encoding could ameliorate some of these structural weaknesses.

Future research should focus on architectural interventions that directly address these biases, as well as systematic probing of generative models along the same axes.

Conclusion

This comprehensive bias-centric evaluation, enabled by ASOB-Bench, establishes that diffusion classifiers—despite sharing the text encoding of CLIP—exhibit less attribute misbinding but are fundamentally vulnerable to spatial and contextual shortcuts intrinsic to their reconstruction-error-based inference. These biases are mechanistically grounded in the way model error is aggregated and are consistent across architectural variants. Understanding and counteracting these biases is immediately relevant for both discriminative and generative applications of diffusion models and will be essential for the reliable deployment of multimodal AI systems.

Figure 7: Representative two-object scenes for the five non-color attribute categories, illustrating the generalized evaluation protocol across diverse attribute bindings.