Multi-Rationale Explainable Object Recognition

• The paper introduces a multi-rationale explainable benchmark that extends single rationale evaluation to set prediction for comprehensive object recognition.
• It employs an annotation pipeline using open-vocabulary detectors and human supervision to generate multiple, visible discriminative cues per image.
• The structured metrics (RR, RW, WR, WW) and the Contrastive Conditional Inference (CCI) method demonstrate notable improvements over prompt-based DROR and ECOR baselines.

Searching arXiv for the benchmark paper and closely related antecedents to ground the article in the cited literature. The Multi-Rationale Explainable Object Recognition Benchmark is a benchmark for evaluating vision-LLMs on joint category prediction and human-readable justification when multiple rationales apply to a single image. Introduced as an extension of earlier explainable object recognition settings, it reformulates explainability from a single-rationale decision problem into a set prediction problem over rationales: given an image, a model must predict the category and a set of rationales that justify the decision. In the benchmark’s formulation, each sample is represented as $(x,c,r)$ or, in the multi-rationale extension, effectively as $(x,c,r_1,r_2,\dots,r_M)$, where $x$ is an image, $c$ is a category, and $r_i$ are rationales describing discriminative attributes that justify the category (Rasekh et al., 19 Aug 2025).

1. Conceptual basis and motivation

The benchmark was introduced to address two limitations attributed to prior explainable object recognition settings. First, earlier datasets such as DROR assigned only one rationale per image, even though the underlying object categories often admit several discriminative attributes. Second, prior methods relied on prompt-based conditioning with templates such as “A photo of a <category> because there is <rationale>,” which the benchmark paper argues does not model conditioning properly in CLIP’s text encoder, especially for multi-rationale inputs (Rasekh et al., 19 Aug 2025).

Within this framework, explainability is not treated as a post hoc visualization or a single textual justification. It is instead formalized as the recovery of a set of explanatory cues that jointly support category recognition. The benchmark therefore supports two tasks. The first is multi-rationale retrieval / prediction, in which the model must identify the correct category $c$ and the set of rationales $r_1,\dots,r_M$. The second is conditional explainable classification, written as

$P(c \mid x, r_1, r_2, \dots, r_M),$

with the full inference objective

$$\max_{c,r_1,\dots,r_M} P(c,r_1,r_2,\dots,r_M \mid x) \equiv \max_{c,r_1,\dots,r_M} P(r_1,r_2,\dots,r_M \mid x)\, P(c \mid r_1,r_2,\dots,r_M,x).$$

This formulation makes the benchmark structurally different from single-rationale classification. The central object of evaluation is no longer a single correct explanatory phrase, but a rationale set whose elements may each capture distinct visible evidence. A plausible implication is that the benchmark is designed to test whether a model can aggregate heterogeneous discriminative cues rather than merely retrieve one salient attribute.

2. Antecedents in explainable object recognition and explanation multiplicity

The immediate precursor is doubly right object recognition (DROR), which proposed that a model should be counted as fully correct only when it gets both the category prediction and the rationale prediction right. DROR constructed datasets in which each image had a category label and one or more visual rationales, and operationalized evaluation through the four cases RR, RW, WR, and WW, with RR denoting “right classification, right rationale” (Mao et al., 2022). DROR’s empirical finding was that state-of-the-art visual models such as CLIP often achieved the correct label while supplying an incorrect rationale, and the paper notes that the best CLIP-H/14 model gets less than 1% doubly right accuracy on ImageNet+. Its proposed “why prompt” improved doubly right performance by up to 28 points and yielded up to 12 points gain in zero-shot transfer (Mao et al., 2022).

The second relevant antecedent is ECOR, which is explicitly described as not a new benchmark but as a method for explainable object recognition built on CLIP and evaluated on the existing DROR datasets. ECOR defined explainability probabilistically through the joint event

$P(c,\{r_i\}_{i=1}^{m}\mid I),$

and selected the rationale-first factorization

$(x,c,r_1,r_2,\dots,r_M)$0

However, despite this general notation, the actual DROR data used in ECOR assigned each image one category and one rationale, so the empirical benchmark remained single-rationale-per-image (Rasekh et al., 2024).

A broader conceptual precursor comes from work on multiple explanations for image classifiers. The paper introducing MultEX argued that many images admit several different sufficient subsets of pixels that all support the same class prediction, and reported that around 96% of images have multiple disjoint explanations under its method. It also showed that, given one explanation, finding a different explanation is DP-complete, that the maximum number of explanations can be exponentially large with

$(x,c,r_1,r_2,\dots,r_M)$1

and that selecting a subset of explanations with bounded overlap is NP-hard (Chockler et al., 2023). Although MultEX is not an object recognition benchmark, it provides a direct rationale for why a single “correct” explanation can be theoretically and empirically inadequate.

3. Dataset composition and annotation pipeline

The benchmark extends the earlier DROR and ECOR explainable object recognition setups from single-rationale to multi-rationale evaluation. It contains six dataset variants: CIFAR-10†, CIFAR-100†, Food-101†, Caltech-101†, SUN†, and ImageNet† (Rasekh et al., 19 Aug 2025).

Its dataset statistics are reported as follows:

• CIFAR-10†: 1,773 images, 10 categories, 56 rationales, average rationales per image ~3.5.
• CIFAR-100†: 13,832 images, 98 categories, 459 rationales, average rationales per image ~3.1.
• Food-101†: 12,676 images, 90 categories, 393 rationales, average rationales per image ~3.4.
• Caltech-101†: 13,035 images, 96 categories, 451 rationales, average rationales per image ~2.9.
• SUN†: 53,975 images, 396 categories, 1,694 rationales, average rationales per image ~3.3.
• ImageNet†: 218,420 images, 994 categories, 4,158 rationales, average rationales per image ~2.9 (Rasekh et al., 19 Aug 2025).

The annotation pipeline begins from the DROR datasets, where each image originally had about 7 rationales per category, but only one rationale was assigned to each image. To create multi-rationale annotations, an open-vocabulary object detector is run on the image; candidate rationales are retained if detector confidence is above 0.1; the process is carried out under human supervision to reduce detector mistakes and bias; and the resulting data yield an average of about 3 rationales per image (Rasekh et al., 19 Aug 2025).

This pipeline inherits the earlier DROR rationale source. In DROR, rationales were generated by asking GPT-3, “What are useful visual features for distinguishing a $(x,c,r_1,r_2,\dots,r_M)$2 in a photo?”, collecting GPT-generated attributes, and then retrieving aligned images using Google Image Search with class-plus-attribute queries (Mao et al., 2022). In the multi-rationale benchmark, the role of the new annotation stage is therefore not to create arbitrary duplicates, but to identify multiple visible attributes per image from a pre-existing rationale vocabulary. This suggests a shift from category-level rationale inventories to image-level rationale sets.

4. Formal task structure and evaluation protocol

The benchmark evaluates each image using four mutually exclusive outcome types: RR, RW, WR, and WW, with the goal of maximizing RR (Rasekh et al., 19 Aug 2025). In the multi-rationale setting, rationale correctness is no longer binary at the image level. Instead, the benchmark defines

$(x,c,r_1,r_2,\dots,r_M)$3

Using this rationale accuracy, the benchmark defines the four metrics as

$(x,c,r_1,r_2,\dots,r_M)$4

$(x,c,r_1,r_2,\dots,r_M)$5

$(x,c,r_1,r_2,\dots,r_M)$6

$(x,c,r_1,r_2,\dots,r_M)$7

The four values sum to 100% (Rasekh et al., 19 Aug 2025).

These definitions generalize the single-rationale DROR evaluation logic. In the original doubly right formulation, the benchmark asked whether the model was “right for the right reasons,” and categorized predictions into RR, RW, WR, and WW according to whether the class and rationale were correct (Mao et al., 2022). The multi-rationale benchmark preserves the same decomposition but replaces binary rationale correctness with a proportion over the set of ground-truth rationales. As a consequence, RR becomes a graded measure of success rather than a strictly all-or-nothing event.

The benchmark paper argues that such decomposition is necessary because a model may get the category right while missing most rationales, or may identify good rationales but still fail on the label. A plausible implication is that this protocol distinguishes between explanation incompleteness and outright explanation failure more precisely than single-score accuracy.

5. Methods evaluated on the benchmark and reported empirical behavior

The benchmark paper evaluates a proposed method, Contrastive Conditional Inference (CCI), against DROR and ECOR baselines. Both baselines are described as relying on prompt-based conditioning with “because,” whereas CCI is introduced specifically to avoid prompt-based conditioning and to model the relationships among image embeddings, rationales, and category embeddings more explicitly (Rasekh et al., 19 Aug 2025).

CCI’s motivation is formalized through a Bayes-style consistency ratio,

$(x,c,r_1,r_2,\dots,r_M)$8

The benchmark paper states that if conditioning were modeled correctly, $(x,c,r_1,r_2,\dots,r_M)$9 should be close to 1; in experiments, the “because” prompt gives values far from 1, while CCI is close to 1 on several datasets (Rasekh et al., 19 Aug 2025).

For zero-shot evaluation on the multi-rationale benchmark, the paper reports that CCI outperforms both baselines on the primary metric RR across all datasets except Caltech-101, where it ranks second. Representative results are:

• CIFAR-10†: CCI RR 41.18, DROR 36.94, ECOR 34.02.
• CIFAR-100†: CCI RR 22.56, DROR 17.99, ECOR 21.41.
• Food-101†: CCI RR 22.15, DROR 15.21, ECOR 21.88.
• ImageNet†: CCI RR 8.35, DROR 2.92, ECOR 4.83 (Rasekh et al., 19 Aug 2025).

For fine-tuned evaluation, the reported CCI results are:

• CIFAR-10†: RR 58.92.
• CIFAR-100†: RR 28.48.
• Food-101†: RR 26.26.
• Caltech-101†: RR 28.80.
• ImageNet†: RR 12.61 (Rasekh et al., 19 Aug 2025).

The benchmark paper emphasizes that DROR and ECOR struggle because they optimize each rationale independently and are not designed for multi-rationale joint inference. It also reports cross-dataset evaluation, concluding that CCI remains strong across domain shifts, shows only mild performance degradation, and outperforms DROR and ECOR in both zero-shot and fine-tuned settings (Rasekh et al., 19 Aug 2025).

These results situate the benchmark as more than a static dataset collection. It functions as an evaluative stress test for whether rationale-conditioned inference is genuinely joint. That emphasis also clarifies the distinction from ECOR: although ECOR’s notation allowed $x$0, its empirical validation remained on single-rationale DROR datasets, and it did not provide a multi-rationale benchmark or dataset construction (Rasekh et al., 2024).

6. Interpretation, limitations, and significance for future research

The benchmark’s principal significance lies in its explicit claim that explainable object recognition should be evaluated as a multi-rationale problem rather than a single-rationale one (Rasekh et al., 19 Aug 2025). This position is consistent with the broader causal view that multiple explanations are not competing answers in the sense of one being correct and the others wrong, but can instead function as alternative sufficient supports for the same prediction (Chockler et al., 2023). In object recognition, that perspective translates into different visible attributes of the same object jointly or separately supporting a class label.

The benchmark paper also states several limitations. The benchmark is built from CLIP-style contrastive models, and the CCI framework is tied to that embedding geometry. The rationale selection step can be computationally challenging. The current benchmark relies on detected rationales plus human supervision, and therefore still depends on detector quality (Rasekh et al., 19 Aug 2025). These constraints indicate that the benchmark is not purely a human-annotated ground-truth resource, nor is it architecturally neutral in the strongest possible sense.

At the same time, the benchmark broadens the problem definition inherited from DROR. DROR established that a model can be right for the wrong reason and that benchmark design must evaluate labels together with rationales (Mao et al., 2022). The multi-rationale extension preserves that normative structure while replacing single-attribute correctness with rationale-set evaluation. A plausible implication is that future benchmark development may need to address not only rationale correctness and category correctness, but also richer issues such as rationale interaction, redundancy, and the distinction between jointly necessary and merely individually sufficient evidence.

The benchmark paper presents its contribution in exactly those terms: it expands DROR and ECOR from one rationale per image to several rationales per image, defines RR/RW/WR/WW for the multi-rationale setting, and uses the resulting framework to argue that prompt-based conditioning is insufficient for robust rationale-conditioned classification (Rasekh et al., 19 Aug 2025). Within the explainable object recognition literature, it therefore marks the point at which the benchmarked task becomes explicitly set-valued on the explanation side rather than scalar.