Contrastive Conditional Inference (CCI)

• Contrastive Conditional Inference (CCI) is a framework that jointly selects object categories and multiple rationales using a probabilistic and geometric method in CLIP space.
• It improves over prompt-based methods by explicitly modeling both rationale selection and category disambiguation through contrastive conditional scoring.
• CCI achieves state-of-the-art zero-shot and fine-tuned performance on multi-rationale benchmarks, validating its effectiveness in explainable object recognition.

Searching arXiv for the cited CCI paper and closely related uses of the term. arxiv_search query: "^^^^^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^^^^^"Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference\""

Contrastive Conditional Inference (CCI) is a training-free framework for explainable object recognition with vision-LLMs such as CLIP. In the formulation introduced in "Multi-Rationale Explainable Object Recognition via Contrastive Conditional Inference" (&&&^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^&&&), the task is not only to predict an accurate object category, but to do so together with rationales that justify the decision-making process. The framework addresses two limitations identified in prior work: prompt-based conditioning suffers from limitations in CLIP’s text encoder and provides weak conditioning on explanatory structures, and existing datasets are often restricted to single, frequently noisy, rationales. CCI therefore combines a multi-rationale benchmark with an explicit probabilistic model over image embeddings, category labels, and rationale embeddings, yielding strong zero-shot performance and state-of-the-art results on the proposed benchmark.

1. Formal definition and inference objective

CCI assumes an input image embedding PRESERVED_PLACEHOLDER_^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^, already produced by CLIP’s image encoder and normalized onto the unit sphere; a finite set of candidate rationales $$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$, embedded by CLIP’s text encoder; and a finite set of object category embeddings $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$, likewise on the unit sphere (&&&^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^&&&). The central goal is to choose a subset of $M$ rationales and an object category that jointly maximize

$$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$

Conditioning on an observed image embedding yields the factorization

$P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$

where $R = \{r_{i_1}, \dots, r_{i_M}\}$. The first factor scores how likely a rationale set is given the image, while the second scores how well a candidate category fits the image under those rationales. In this sense, CCI treats explainable recognition as joint conditional inference over semantic evidence and class identity rather than as prompt completion or template matching.

This probabilistic decomposition is the defining feature of the method. It separates rationale selection from category disambiguation while still coupling them in the final objective. A plausible implication is that CCI can represent explanatory uncertainty more explicitly than methods that collapse rationales into a single textual prompt.

2. Contrastive conditional scoring in CLIP space

CCI models both factors contrastively. For rationale selection, the framework uses the rationale-selection prior

$$P(r \mid x) = \frac{\exp\!\bigl(\tau (x \cdot r)\bigr)} {\sum_{r' \in \mathcal{R}} \exp\!\bigl(\tau (x \cdot r')\bigr)},$$

with temperature $\tau > 0$. If an ordered set of $M$ rationales is required, the method can select them sequentially or via beam search.

Given a chosen rationale set $$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^, CCI defines a conditional compatibility score

$$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$1

The construction of this score is geometric. Let

$$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$2

The projection of a candidate category embedding $$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$3 onto the span of $$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$4 is computed by

$$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$5

The “desirable direction” is the uniform average

$$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$6

Categories are then normalized through

$$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$7

If the objective is only to pick the best category for a given image and rationale set, the final decision rule is

$$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$8

This makes the framework fully training-free at inference time while still imposing structured conditioning through a probabilistic score.

3. Geometric interpretation and relation to prompt-based methods

CCI departs from prompt-based explainable recognition by replacing textual templates such as “A photo of a $$\mathcal{R} = \{r_1, r_2, \dots, r_{|\mathcal{R}|}\} \subset \mathbb{R}^d$$9 because there is $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^” with an explicit linear subspace construction (&&&^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^&&&). The framework defines

$$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$1

described as the linear subspace or hyperplane generated by the image embedding and the selected rationale embeddings.

Under the assumption that a correct category embedding should lie close to both the image and all its ground-truth rationales, the optimal category must lie in this hyperplane. The inference procedure therefore has two steps: first project each candidate category onto the subspace, then measure how well that projection aligns with the positive average direction $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$2. By Theorem 2 in the paper, this direction best aligns with both image and rationales.

The resulting interpretation is that CCI conditions on rationales by changing the admissible geometric region for category embeddings, not by asking CLIP’s text encoder to express explanatory structure in natural language. This directly addresses the paper’s claim that “because” conditioning is poor in CLIP’s text encoder. A common misconception is therefore to view CCI as another prompt-engineering strategy; in the paper’s formulation, it is instead a contrastive, projection-based conditional inference rule operating in embedding space.

4. Multi-rationale benchmark and evaluation protocol

The same work introduces a multi-rationale explainable object recognition benchmark designed to replace the single-rationale setting inherited from prior datasets (&&&^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^&&&). Starting from the single-rationale DROR splits on CIFAR-1^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^, CIFAR-1^^^^1(Rasekh et al., 19 Aug 2025) OR \1(Rasekh et al., 19 Aug 2025) OR \1^^^^, Food-1^^^^1(Rasekh et al., 19 Aug 2025) OR \11^^^^, Caltech-1^^^^1(Rasekh et al., 19 Aug 2025) OR \11^^^^, SUN and ImageNet, the authors ran an open-vocabulary object detector on each image, kept all detections above confidence $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$3, and obtained on average 3–4 rationales per image. The resulting datasets are denoted, for example, CIFAR-1^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^C={c1,c2,…,cN}⊂Rd\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d4 and CIFAR-1^^^^1(Rasekh et al., 19 Aug 2025) OR \1(Rasekh et al., 19 Aug 2025) OR \1^^^^C={c1,c2,…,cN}⊂Rd\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d5.

For each test image with ground-truth category $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$6 and ground-truth rationale set $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$7, a method predicts $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$8 and $$\mathcal{C} = \{c_1, c_2, \dots, c_N\} \subset \mathbb{R}^d$$9. Rationale accuracy is defined as

$M$^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^

and category correctness is recorded by

$M$1

Each example is then decomposed into four outcome types:

$M$2

$M$3

$M$4

$M$5

Since

$M$6

the primary goal is to maximize $M$7 on the test set.

This metric design is important because it distinguishes correct explanations for correct predictions from partially correct or fully incorrect predictions. In particular, it prevents a method from being judged only by classification accuracy when the explanatory rationales are wrong, or only by rationale overlap when the category prediction fails.

5. Reported empirical performance

On the multi-rationale benchmark, CCI is reported to achieve state-of-the-art results, including strong zero-shot performance, and to set a new standard for both classification accuracy and rationale quality (&&&^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^&&&). The paper’s condensed results table gives representative zero-shot (“ZS”) and fine-tuned (“FT”) $M$8 values for CIFAR-1^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^MM9 and ImageNet$$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^.

For CIFAR-1^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^P(c,ri1,…,riM∣x)=P(ri1,…,riM∣x)  P(c∣x,ri1,…,riM).P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .1, DROR reports $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$2 in the zero-shot setting and $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$3 after fine-tuning, while ECOR reports $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$4 and $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$5. CCI reports $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$6 zero-shot and $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$7 fine-tuned. For ImageNet$$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$8, DROR reports $$P(c, r_{i_1}, \ldots, r_{i_M} \mid x) = P(r_{i_1}, \dots, r_{i_M} \mid x)\; P(c \mid x, r_{i_1}, \dots, r_{i_M}) .$$9 zero-shot and $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^ fine-tuned, ECOR reports $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$1 and $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$2, and CCI reports $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$3 and $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$4.

Across all six datasets, the paper states that CCI achieves the highest zero-shot $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$5 in five out of six cases, often by a large margin, and that fine-tuning further narrows the gap. Because $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$6 is explicitly defined as the primary objective, these results are presented as evidence that the method improves the joint target of correct categories with correct rationales rather than optimizing only one side of the explainability problem.

6. Strengths, limitations, and terminological scope

The paper identifies three principal strengths of CCI: it is fully training-free in the zero-shot setting while conditioning on multiple rationales through a principled probabilistic and geometric treatment; it explicitly models $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$7 in CLIP’s embedding space, thereby overcoming weak “because” conditioning in CLIP’s text encoder; and it shows strong transfer and cross-dataset generalization without dataset-specific training (&&&^^^^1(Rasekh et al., 19 Aug 2025) OR \1^^^^&&&).

The limitations are equally explicit. The method is currently tied to contrastive models with fixed embeddings such as CLIP. Extending it to generative or attention-based vision-LLMs such as LLaVA remains open. The uniform weighting in $P(c, R \mid x) = P(R \mid x)\; P(c \mid x, R),$8 may be suboptimal if some rationales are more discriminative, so learning or estimating per-rationale weights is identified as a natural next step. In addition, rationale selection by greedy or beam search may be improved by global combinatorial optimization, though at the cost of tractability.

The term “contrastive conditional” also appears in other arXiv works, but in substantially different senses. "Information-theoretic stochastic contrastive conditional GAN: InfoSCC-GAN" frames conditional generation through an unsupervised contrastive encoder, an attribute classifier, and an EigenGAN generator, with an information-theoretic interpretation of the training objectives (Kinakh et al., 2021). "Contrastive Conditional Neural Processes" augments conditional neural processes with two hierarchical contrastive branches, temporal contrastive learning (TCL) and function contrastive learning (FCL), to improve function distribution reconstruction and parameter identification (Ye et al., 2022). This suggests that the CCI of explainable object recognition is best understood as a specific probabilistic-geometric inference framework rather than as a universal label for all contrastive conditional methods.