---
title: Multi-Rationale Explainable Object Recognition
url: https://www.emergentmind.com/topics/multi-rationale-explainable-object-recognition-benchmark
type: topic
---

# Multi-Rationale Explainable Object Recognition

Searching arXiv for the benchmark paper and closely related antecedents to ground the article in the cited literature.
The **Multi-Rationale Explainable Object Recognition Benchmark** is a benchmark for evaluating vision-language models on **joint category prediction and human-readable justification** when **multiple rationales** apply to a single image. Introduced as an extension of earlier explainable object recognition settings, it reformulates explainability from a single-rationale decision problem into a **set prediction problem** over rationales: given an image, a model must predict the category and a set of rationales that justify the decision. In the benchmark’s formulation, each sample is represented as \((x,c,r)\) or, in the multi-rationale extension, effectively as \((x,c,r_1,r_2,\dots,r_M)\), where \(x\) is an image, \(c\) is a category, and \(r_i\) are rationales describing discriminative attributes that justify the category [2508.14280].

## 1. Conceptual basis and motivation

The benchmark was introduced to address two limitations attributed to prior explainable object recognition settings. First, earlier datasets such as DROR assigned **only one rationale per image**, even though the underlying object categories often admit several discriminative attributes. Second, prior methods relied on prompt-based conditioning with templates such as “A photo of a <category> because there is <rationale>,” which the benchmark paper argues does not model conditioning properly in CLIP’s text encoder, especially for multi-rationale inputs [2508.14280].

Within this framework, explainability is not treated as a post hoc visualization or a single textual justification. It is instead formalized as the recovery of a set of explanatory cues that jointly support category recognition. The benchmark therefore supports two tasks. The first is **multi-rationale retrieval / prediction**, in which the model must identify the correct category \(c\) and the set of rationales \(r_1,\dots,r_M\). The second is **conditional explainable classification**, written as
\[
P(c \mid x, r_1, r_2, \dots, r_M),
\]
with the full inference objective
\[
\max_{c,r_1,\dots,r_M} P(c,r_1,r_2,\dots,r_M \mid x) \equiv \max_{c,r_1,\dots,r_M} P(r_1,r_2,\dots,r_M \mid x)\, P(c \mid r_1,r_2,\dots,r_M,x).
\]

This formulation makes the benchmark structurally different from single-rationale classification. The central object of evaluation is no longer a single correct explanatory phrase, but a rationale set whose elements may each capture distinct visible evidence. A plausible implication is that the benchmark is designed to test whether a model can aggregate heterogeneous discriminative cues rather than merely retrieve one salient attribute.

## 2. Antecedents in explainable object recognition and explanation multiplicity

The immediate precursor is **doubly right object recognition** (DROR), which proposed that a model should be counted as fully correct only when it gets **both** the category prediction and the rationale prediction right. DROR constructed datasets in which each image had a **category label** and one or more **visual rationales**, and operationalized evaluation through the four cases **RR**, **RW**, **WR**, and **WW**, with **RR** denoting “right classification, right rationale” [2212.06202]. DROR’s empirical finding was that state-of-the-art visual models such as CLIP often achieved the correct label while supplying an incorrect rationale, and the paper notes that the best CLIP-H/14 model gets **less than 1% doubly right accuracy on ImageNet+**. Its proposed “why prompt” improved doubly right performance by up to **28 points** and yielded up to **12 points** gain in zero-shot transfer [2212.06202].

The second relevant antecedent is **ECOR**, which is explicitly described as **not a new benchmark** but as a method for explainable object recognition built on CLIP and evaluated on the existing DROR datasets. ECOR defined explainability probabilistically through the joint event
\[
P(c,\{r_i\}_{i=1}^{m}\mid I),
\]
and selected the **rationale-first factorization**
\[
P(c,\{r_i\}_{i=1}^{m}\mid I)=P(\{r_i\}_{i=1}^{m}\mid I)P(c\mid \{r_i\}_{i=1}^{m},I).
\]
However, despite this general notation, the actual DROR data used in ECOR assigned each image **one category** and **one rationale**, so the empirical benchmark remained **single-rationale-per-image** [2404.12839].

A broader conceptual precursor comes from work on multiple explanations for image classifiers. The paper introducing **MultEX** argued that many images admit **several different sufficient subsets** of pixels that all support the same class prediction, and reported that around **96%** of images have multiple disjoint explanations under its method. It also showed that, given one explanation, finding a different explanation is **DP-complete**, that the maximum number of explanations can be exponentially large with
\[
\#\text{ explanations} \le \binom{n}{\lfloor n/2 \rfloor},
\]
and that selecting a subset of explanations with bounded overlap is **NP-hard** [2309.14309]. Although MultEX is not an object recognition benchmark, it provides a direct rationale for why a single “correct” explanation can be theoretically and empirically inadequate.

## 3. Dataset composition and annotation pipeline

The benchmark extends the earlier **DROR** and **ECOR** explainable object recognition setups from **single-rationale** to **multi-rationale** evaluation. It contains six dataset variants: **CIFAR-10†**, **CIFAR-100†**, **Food-101†**, **Caltech-101†**, **SUN†**, and **ImageNet†** [2508.14280].

Its dataset statistics are reported as follows:

- **CIFAR-10†**: **1,773** images, **10** categories, **56** rationales, average rationales per image **~3.5**.
- **CIFAR-100†**: **13,832** images, **98** categories, **459** rationales, average rationales per image **~3.1**.
- **Food-101†**: **12,676** images, **90** categories, **393** rationales, average rationales per image **~3.4**.
- **Caltech-101†**: **13,035** images, **96** categories, **451** rationales, average rationales per image **~2.9**.
- **SUN†**: **53,975** images, **396** categories, **1,694** rationales, average rationales per image **~3.3**.
- **ImageNet†**: **218,420** images, **994** categories, **4,158** rationales, average rationales per image **~2.9** [2508.14280].

The annotation pipeline begins from the DROR datasets, where each image originally had about **7 rationales per category**, but only one rationale was assigned to each image. To create multi-rationale annotations, an **open-vocabulary object detector** is run on the image; candidate rationales are retained if detector confidence is **above 0.1**; the process is carried out **under human supervision** to reduce detector mistakes and bias; and the resulting data yield an average of about **3 rationales per image** [2508.14280].

This pipeline inherits the earlier DROR rationale source. In DROR, rationales were generated by asking GPT-3, “What are useful visual features for distinguishing a \(\{category\}\) in a photo?”, collecting GPT-generated attributes, and then retrieving aligned images using Google Image Search with class-plus-attribute queries [2212.06202]. In the multi-rationale benchmark, the role of the new annotation stage is therefore not to create arbitrary duplicates, but to identify multiple visible attributes per image from a pre-existing rationale vocabulary. This suggests a shift from category-level rationale inventories to image-level rationale sets.

## 4. Formal task structure and evaluation protocol

The benchmark evaluates each image using **four mutually exclusive outcome types**: **RR**, **RW**, **WR**, and **WW**, with the goal of maximizing **RR** [2508.14280]. In the multi-rationale setting, rationale correctness is no longer binary at the image level. Instead, the benchmark defines
\[
\text{accuracy}_{\text{rationales}}=\frac{|\text{correct rationales}|}{|\text{ground-truth rationales}|}.
\]

Using this rationale accuracy, the benchmark defines the four metrics as
\[
RR \in [0,1] = 1(\text{category is right}) \cdot \text{accuracy}_{\text{rationales}},
\]
\[
RW \in [0,1] = 1(\text{category is right}) \cdot (1-\text{accuracy}_{\text{rationales}}),
\]
\[
WR \in [0,1] = 1(\text{category is wrong}) \cdot \text{accuracy}_{\text{rationales}},
\]
\[
WW \in [0,1] = 1(\text{category is wrong}) \cdot (1-\text{accuracy}_{\text{rationales}}).
\]
The four values sum to **100%** [2508.14280].

These definitions generalize the single-rationale DROR evaluation logic. In the original doubly right formulation, the benchmark asked whether the model was “right for the right reasons,” and categorized predictions into **RR**, **RW**, **WR**, and **WW** according to whether the class and rationale were correct [2212.06202]. The multi-rationale benchmark preserves the same decomposition but replaces binary rationale correctness with a proportion over the set of ground-truth rationales. As a consequence, **RR** becomes a graded measure of success rather than a strictly all-or-nothing event.

The benchmark paper argues that such decomposition is necessary because a model may get the category right while missing most rationales, or may identify good rationales but still fail on the label. A plausible implication is that this protocol distinguishes between explanation incompleteness and outright explanation failure more precisely than single-score accuracy.

## 5. Methods evaluated on the benchmark and reported empirical behavior

The benchmark paper evaluates a proposed method, **Contrastive Conditional Inference (CCI)**, against **DROR** and **ECOR** baselines. Both baselines are described as relying on prompt-based conditioning with “because,” whereas CCI is introduced specifically to avoid prompt-based conditioning and to model the relationships among image embeddings, rationales, and category embeddings more explicitly [2508.14280].

CCI’s motivation is formalized through a Bayes-style consistency ratio,
\[
RT = \frac{P(\hat{c}|x)P(\hat{r}|\hat{c},x)}{P(\hat{r}|x)P(\hat{c}|\hat{r},x)}.
\]
The benchmark paper states that if conditioning were modeled correctly, \(RT\) should be close to 1; in experiments, the “because” prompt gives values far from 1, while CCI is close to 1 on several datasets [2508.14280].

For zero-shot evaluation on the multi-rationale benchmark, the paper reports that CCI outperforms both baselines on the primary metric **RR** across all datasets except **Caltech-101**, where it ranks second. Representative results are:

- **CIFAR-10†**: CCI RR **41.18**, DROR **36.94**, ECOR **34.02**.
- **CIFAR-100†**: CCI RR **22.56**, DROR **17.99**, ECOR **21.41**.
- **Food-101†**: CCI RR **22.15**, DROR **15.21**, ECOR **21.88**.
- **ImageNet†**: CCI RR **8.35**, DROR **2.92**, ECOR **4.83** [2508.14280].

For fine-tuned evaluation, the reported CCI results are:

- **CIFAR-10†**: RR **58.92**.
- **CIFAR-100†**: RR **28.48**.
- **Food-101†**: RR **26.26**.
- **Caltech-101†**: RR **28.80**.
- **ImageNet†**: RR **12.61** [2508.14280].

The benchmark paper emphasizes that **DROR and ECOR struggle because they optimize each rationale independently and are not designed for multi-rationale joint inference**. It also reports cross-dataset evaluation, concluding that CCI remains strong across domain shifts, shows only mild performance degradation, and outperforms DROR and ECOR in both zero-shot and fine-tuned settings [2508.14280].

These results situate the benchmark as more than a static dataset collection. It functions as an evaluative stress test for whether rationale-conditioned inference is genuinely joint. That emphasis also clarifies the distinction from ECOR: although ECOR’s notation allowed \(\{r_i\}_{i=1}^{m}\), its empirical validation remained on **single-rationale DROR datasets**, and it did **not** provide a multi-rationale benchmark or dataset construction [2404.12839].

## 6. Interpretation, limitations, and significance for future research

The benchmark’s principal significance lies in its explicit claim that explainable object recognition should be evaluated as a **multi-rationale** problem rather than a single-rationale one [2508.14280]. This position is consistent with the broader causal view that multiple explanations are not competing answers in the sense of one being correct and the others wrong, but can instead function as alternative sufficient supports for the same prediction [2309.14309]. In object recognition, that perspective translates into different visible attributes of the same object jointly or separately supporting a class label.

The benchmark paper also states several limitations. The benchmark is built from **CLIP-style contrastive models**, and the CCI framework is tied to that embedding geometry. The rationale selection step can be **computationally challenging**. The current benchmark relies on **detected rationales plus human supervision**, and therefore still depends on **detector quality** [2508.14280]. These constraints indicate that the benchmark is not purely a human-annotated ground-truth resource, nor is it architecturally neutral in the strongest possible sense.

At the same time, the benchmark broadens the problem definition inherited from DROR. DROR established that a model can be right for the wrong reason and that benchmark design must evaluate labels together with rationales [2212.06202]. The multi-rationale extension preserves that normative structure while replacing single-attribute correctness with rationale-set evaluation. A plausible implication is that future benchmark development may need to address not only rationale correctness and category correctness, but also richer issues such as rationale interaction, redundancy, and the distinction between jointly necessary and merely individually sufficient evidence.

The benchmark paper presents its contribution in exactly those terms: it expands DROR and ECOR from one rationale per image to several rationales per image, defines RR/RW/WR/WW for the multi-rationale setting, and uses the resulting framework to argue that prompt-based conditioning is insufficient for robust rationale-conditioned classification [2508.14280]. Within the explainable object recognition literature, it therefore marks the point at which the benchmarked task becomes explicitly set-valued on the explanation side rather than scalar.

Source: https://www.emergentmind.com/topics/multi-rationale-explainable-object-recognition-benchmark