SUB Benchmark: CBMs Under Attribute Shift

• SUB is a fine-grained benchmark that isolates single-attribute distribution shifts to evaluate whether CBMs genuinely ground their concept predictions.
• The dataset uses synthetic bird images generated with Tied Diffusion Guidance to substitute specific attributes while preserving overall class identity.
• Empirical results reveal that high in-distribution concept accuracy does not ensure robust attribute grounding, highlighting a gap in current CBM approaches.

Searching arXiv for the SUB benchmark and related CBM robustness work. SUB is a fine-grained image and concept benchmark designed to evaluate Concept Bottleneck Models (CBMs) and related concept-based interpretable models under single-attribute distribution shifts. It consists of synthetic bird images derived from CUB, constructed so that a specific concept such as wing color or belly pattern is substituted while the rest of the bird remains visually consistent with a reference class. The benchmark was introduced to test whether concept predictors are genuinely grounded in visual evidence, rather than merely recovering memorized class-prototype concept vectors, and is paired with a generation method called Tied Diffusion Guidance (TDG) to control both class identity and attribute substitution (Bader et al., 31 Jul 2025).

1. Motivation and problem setting

Deep concept-based models such as Concept Bottleneck Models predict a set of human-defined concepts and then predict a class label from those concepts. In settings such as medicine, the intended advantage is not only predictive performance but also interpretable reasoning: when a model predicts a concept, that prediction is expected to reflect the input rather than a statistical shortcut (Bader et al., 31 Jul 2025).

The motivating concern behind SUB is that prior CBM evaluations were limited to the same distribution of training classes and concept vectors. Real-world deployment, however, involves distributional shifts and novel combinations of concepts. The benchmark was therefore constructed to diagnose whether CBMs truly ground their concept predictions in the image when a single attribute is perturbed. Its design target is explicitly fine-grained: it isolates individual attribute changes while keeping the broader class identity intact.

This setup addresses a common misconception in concept-based interpretability. Strong in-distribution concept accuracy does not by itself establish that a model is visually grounded. SUB operationalizes this distinction by holding most of the bird constant while modifying one attribute, thereby separating class memorization from concept recognition. This suggests that interpretability claims based solely on same-distribution concept metrics are incomplete.

2. Dataset construction and curation pipeline

SUB is derived from the CUB–200–2011 bird dataset, which contains 200 species annotated with 312 binary part-and-attribute labels such as “blue crown” and “spotted belly.” To ensure reliable generation and filtering, the construction pipeline first rendered 20 samples per class with a frozen diffusion model, FLUX.1-dev, and retained the 33 species that a pre-trained CUB classifier recognized with 100 % accuracy (Bader et al., 31 Jul 2025).

The 312 binary attributes were grouped into 28 attribute groups, including color, shape, and pattern. A visual-question-answering model was then used to select 45 attributes that it could recognize at least 90 % of the time. For each of the 33 retained bird classes, all 45 attributes not originally possessed by that bird were selected as substitution targets, yielding 1 485 candidate $(\text{bird}, \text{attribute})$ pairs.

For each candidate pair, 500 images were generated using TDG. The same VQA model was then used for automatic filtering: images in the bottom 90 % of target-attribute confidence were removed, and candidate pairs with fewer than 50 surviving images were discarded. Human validation followed: four annotators reviewed 40 images for each surviving pair and checked both whether the target attribute was faithfully applied and whether the rest of the bird still resembled the reference. Only pairs with at least 90 % human agreement were retained, and the final dataset was capped at 50 images per pair.

The resulting benchmark contains 768 unique $(\text{bird}, \text{substituted-attribute})$ combinations and 38 400 images. Because SUB uses CUB species and attribute names, existing CBMs trained on CUB can be evaluated on these novel combinations without retraining.

Component	Value	Role
Reference classes	33	CUB species retained after classifier filtering
Candidate concepts	45	VQA-recognizable attributes
Final combinations	768	Surviving $(\text{bird}, \text{attribute})$ pairs
Images per combination	50	Final cap after validation
Total images	38 400	Benchmark size

The construction procedure is notable because it combines automatic filtering and human validation rather than treating synthetic generation alone as sufficient. A plausible implication is that SUB is intended not merely as a generative showcase but as a controlled evaluation substrate for zero-shot robustness testing.

3. Tied Diffusion Guidance (TDG)

TDG is the image-generation mechanism used to construct SUB. It is introduced to control a standard Latent Diffusion Model in a way that preserves the reference bird class while transferring a target attribute from a guidance bird that naturally exhibits that attribute (Bader et al., 31 Jul 2025).

A standard LDM predicts noise

$\epsilon_\theta(x_t,\,c,\,t),$

where $c$ is the text prompt. Under classifier-free guidance, the prediction becomes

$$\tilde\epsilon_\theta(x_t,c,t) = \epsilon_\theta(x_t,t) + s_g\bigl[\epsilon_\theta(x_t,c,t) - \epsilon_\theta(x_t,t)\bigr],$$

so that $s_g$ controls how strongly the prompt steers generation.

TDG generates two images in parallel: a reference image $x^\mathcal R$ of class $\mathcal R$ with one attribute $\mathcal S^-$ replaced by $(\text{bird}, \text{substituted-attribute})$0, and a guidance image $(\text{bird}, \text{substituted-attribute})$1 of class $(\text{bird}, \text{substituted-attribute})$2 which natively exhibits $(\text{bird}, \text{substituted-attribute})$3. The two diffusion trajectories are tied by initializing from the same noise and by sharing predicted noise on pixels where the two prompt-conditioned predictions agree. With

$(\text{bird}, \text{substituted-attribute})$4

TDG defines an element-wise fused noise

$(\text{bird}, \text{substituted-attribute})$5

where $(\text{bird}, \text{substituted-attribute})$6 is the $(\text{bird}, \text{substituted-attribute})$7-th percentile of absolute differences across all pixels. The tied predictions are then

$(\text{bird}, \text{substituted-attribute})$8

The operative idea is that early denoising steps are tightly coupled, while later steps decouple the two images. By sharing noise where the two predictions agree, TDG is intended to ensure that the reference image adopts the target attribute from the guidance image while preserving the remaining class-specific visual structure. Once generation is complete, the guidance image is discarded.

In methodological terms, TDG solves a control problem specific to benchmark construction. The benchmark requires attribute-level intervention without uncontrolled corruption of class identity; TDG provides that intervention by coupling two conditioned diffusion processes rather than relying on a single prompt or naive editing procedure.

4. Evaluation protocol and model families

SUB evaluates how well a CBM trained on original CUB classes detects the newly substituted attribute $(\text{bird}, \text{substituted-attribute})$9 and rejects the removed attribute $(\text{bird}, \text{attribute})$0. The primary metrics are concept accuracy for $(\text{bird}, \text{attribute})$1, defined as the fraction of SUB images where the model correctly predicts the new attribute, and removal accuracy for $(\text{bird}, \text{attribute})$2, defined as the fraction where the model correctly omits the old attribute (Bader et al., 31 Jul 2025).

These SUB-specific metrics are complemented by concept-level accuracy on the held-out CUB test set, measured both for the subset of attributes used in SUB and for all 312 attributes, in order to characterize in-distribution performance. Human performance is also measured using three-way human annotation on a sample of SUB images, establishing a ceiling of approximately 94 % accuracy on $(\text{bird}, \text{attribute})$3. For binary concepts, random chance is 50 %.

The evaluated model families include independent training, where image-to-concept and concept-to-class components are trained separately; joint training, where both are trained end-to-end; per-concept CBM, which uses one binary classifier per attribute; and Concept Embedding Models (CEM), which predict a two-dimensional “present/absent” embedding per concept. Two labeling schemes are compared: “hard” binary labels per image, and “soft” labels derived from annotator confidence or class majority. All models are evaluated zero-shot on SUB without further fine-tuning.

The protocol is carefully aligned with the benchmark’s objective. Because models are not retrained on the new concept combinations, the measured degradation reflects generalization under compositional shift rather than adaptation capacity. This suggests that SUB is best interpreted as a stress test for grounding fidelity, not as a benchmark of transfer learning or synthetic-data finetuning.

5. Empirical results and characteristic failure modes

The central empirical result is that high in-distribution concept accuracy on CUB does not transfer to SUB. Although concept accuracy on CUB exceeds 96 %, all CBM variants fail to detect the substituted attribute $(\text{bird}, \text{attribute})$4 on SUB. The best result is reported for CEM at 45.7 %, which is below the 50 % chance level, while the per-concept CBM collapses to approximately 0.4 % (Bader et al., 31 Jul 2025).

A second finding is that CBMs often maintain or hallucinate the original attribute. Many models achieve more than 85 % “correct” removal of $(\text{bird}, \text{attribute})$5 simply by never switching away from the training concept vector. Thus, removal accuracy is not by itself evidence of successful attribute substitution detection. It can instead reflect persistent prediction of the original class-associated concept profile.

Vision-language-model backbones improve performance only modestly. CLIP and SigLIP/EVA-CLIP extensions perform somewhat better, with EVA-CLIP reaching about 47 % on $(\text{bird}, \text{attribute})$6, but they still fall below random on many attribute groups and hallucinate the original concept at 75–80 %. By contrast, human annotators achieve approximately 94 % on identifying the substituted attribute, supporting the claim that the generated benchmark instances are recognizable to people.

Evaluation target	Reported outcome
In-distribution CUB concept accuracy	$(\text{bird}, \text{attribute})$7
Best SUB concept accuracy on $(\text{bird}, \text{attribute})$8	CEM at 45.7 %
Per-concept CBM on $(\text{bird}, \text{attribute})$9	$\epsilon_\theta(x_t,\,c,\,t),$0
Human accuracy on $\epsilon_\theta(x_t,\,c,\,t),$1	$\epsilon_\theta(x_t,\,c,\,t),$2
Random chance for binary concepts	50 %

The dominant failure mode is described as reliance on memorized class-prototype concept vectors rather than grounding each concept in visual evidence. The canonical example is a “Blue Jay with yellow crown,” where the model recalls the “Blue Jay vector,” including “blue crown,” and ignores the yellow cue. This interpretation is significant because it distinguishes semantic prediction from perceptual grounding: the model may output concepts that are statistically appropriate for the class while being visually incorrect for the instance.

6. Interpretation, limitations, and research implications

SUB exposes a specific brittleness in concept-based interpretability: excellent in-distribution performance does not guarantee faithful concept predictions under minimal distributional shift (Bader et al., 31 Jul 2025). The benchmark therefore reframes the evaluation of CBMs from a question of average concept accuracy to a question of whether individual concepts remain grounded when the input departs from the training concept manifold in a controlled way.

Several practical implications follow directly from the reported findings. First, concept-level explanations based on CBMs should not be assumed reliable under compositional novelty, even when the perturbation is restricted to a single attribute. Second, evaluation suites that do not test such perturbations may overestimate the epistemic value of concept predictions. Third, comparison to human performance matters: the large human-model gap on $\epsilon_\theta(x_t,\,c,\,t),$3 indicates that the benchmark difficulty is not simply a consequence of ambiguous or low-quality synthetic images.

The paper suggests three improvement directions. One is training-time augmentation with synthetic attribute swaps to encourage genuine compositional learning. A second is architectural or regularization changes that force concept classifiers to attend locally to the relevant region, such as the crown patch for crown-color tasks. A third is the use of contrastive or grounding-based objectives that penalize concept predictions when visual evidence disagrees with the concept label. These proposals are not presented as resolved solutions; rather, they indicate where present CBM pipelines appear to fail.

A further point of interpretation concerns benchmark scope. Because SUB is derived from CUB species and attribute names, it is tightly coupled to fine-grained bird recognition and to models trained on that ecosystem. A plausible implication is that the benchmark is most informative as a diagnostic of grounding failure in concept-based classifiers, and only indirectly a proxy for other application domains. Its broader relevance arises from the structure of the test—single-attribute substitution under otherwise preserved identity—rather than from ornithological content per se.

7. Position within concept-based interpretability

Within concept-based interpretability research, SUB functions as a robustness benchmark rather than a new CBM architecture. Its contribution is methodological: it creates a controlled setting in which an interpreter can ask whether a concept prediction tracks the image or the class prior. That distinction is central to any deployment scenario in which concepts are expected to support auditing, error analysis, or safety-critical trust decisions (Bader et al., 31 Jul 2025).

The benchmark’s design makes its diagnostic resolution unusually fine-grained. It does not merely test whether a model generalizes to new classes or broad domain shift; it tests whether the model can revise exactly one concept while holding the remainder of the class description stable. This sharply isolates the tension between compositional generalization and prototype recall.

SUB also clarifies a recurring ambiguity in the evaluation of interpretable models. A system may appear interpretable because it produces human-named concepts with high held-out accuracy, yet still fail the stronger requirement that those concepts be faithfully grounded in the particular image under distribution shift. SUB formalizes this stronger requirement and shows that current CBM variants, including joint, independent, per-concept, and embedding-based versions, do not satisfy it in the reported experiments.