Visual Concept Ranking (VCR)

• Visual Concept Ranking (VCR) is a family of methods that rank candidate responses or images using multimodal evidence from visual and textual inputs.
• It spans diverse tasks—from visual commonsense reasoning with answer/rationale choices to image and concept ranking in educational and auditing contexts.
• Key methodologies involve deep integration of language-vision models, external knowledge injection, dynamic memory, and graph-based reasoning to enhance grounding and performance.

Visual Concept Ranking (VCR) is an overloaded acronym rather than a single standardized task. In the literature on the benchmark introduced by Zellers et al., VCR usually denotes Visual Commonsense Reasoning: given an image and a question, a model ranks four answer candidates and, conditional on the answer, four rationale candidates. In later work, the same initials denote ranking candidate images by how well they depict a concept, ordering discovered concepts in a per-image concept graph by causal influence, or auditing large multimodal models with ranked concept dependencies. Across these usages, VCR remains a ranking problem over multimodal evidence, but the ranked objects, supervision signals, and evaluation criteria differ substantially (Tang et al., 2021, Ji et al., 18 Sep 2025, Hossain et al., 1 Jul 2026, Janizek et al., 4 Feb 2026).

1. Terminological scope and principal meanings

The contemporary literature uses “VCR” in at least four distinct but technically related senses.

Usage of VCR	Representative formulation	Ranked object
Visual Commonsense Reasoning	Multi-choice image-question reasoning	Answer and rationale candidates
Visual Concept Ranking in QuizRank	Image suitability as an illustrative exemplar	Candidate images
Visual Concept Ranking in G-CBM	Per-image causal importance ordering	Discovered concept nodes
Visual Concept Ranking for LMM auditing	Prompt-conditioned concept sensitivity analysis	Textual concept directions

In the Visual Commonsense Reasoning line, the “concepts” being ranked are the answer and rationale candidates. The DMVCR paper states that, in this setting, VCR stands for Visual Commonsense Reasoning and that the ranked items are the answer and rationale candidates; no external concept-level knowledge base is used, although the dictionary $D$ functions as an internal concept store capturing recurring commonsense patterns from training data (Tang et al., 2021). TAB-VCR makes the same point operationally by unifying both subtasks as “choose a response $r$ from four options given a query $q$ and image $I$” (Lin et al., 2019).

QuizRank explicitly distinguishes its use of “Visual Concept Ranking” from the Visual Commonsense Reasoning dataset. There, the task is to produce a ranking over images for a visual concept $C$ according to in-class fidelity and, in the contrastive case, between-class discriminability against a distractor concept $D$ (Ji et al., 18 Sep 2025). By contrast, G-CBM defines VCR as the per-image ordering of discovered visual concepts by their causal influence on the model’s prediction, while the medical auditing paper defines VCR as a ranked list of concepts with effect sizes and statistical significance for a fixed prompt and task (Hossain et al., 1 Jul 2026, Janizek et al., 4 Feb 2026).

This suggests that “Visual Concept Ranking” is best treated as a family of ranking formulations rather than a single benchmark name. The common denominator is not the ontology of the ranked object, but the use of multimodal evidence to order alternatives.

2. Candidate ranking in Visual Commonsense Reasoning

In the benchmark sense, VCR is a multiple-choice visual reasoning task built from movie scenes. Each example includes an image, detected object regions referred to in text by integer tags such as “[0,1],” a natural-language question, four answer choices, and four rationale choices; the dataset provides the correct answer and rationale (Tang et al., 2021). The benchmark contains approximately 110k images and 290k multiple-choice questions, with 290k gold answers and 290k gold rationales, while another description reports over 212k train, 26k validation, and 25k test questions across 110k unique movie scenes (Tang et al., 2021, Lin et al., 2019).

The task is decomposed into three subtasks. Q→A predicts the correct answer $a^*$ from four answer candidates given the image and question. QA→R predicts the correct rationale $r^*$ from four rationale candidates given the image, question, and the correct answer $a^*$. Q→AR is a pipeline metric that counts an item as correct only when both the selected answer and the selected rationale match the gold annotations (Tang et al., 2021). The dataset’s distractor choices are created by adversarial matching, and reported inference types include explanation at 38%, activity at 24%, and the remainder across temporal, mental, role, scene, and hypothetical categories (Tang et al., 2021).

Operationally, benchmark VCR is a ranking problem. For each candidate option, the model forms a query-response pair, computes a compatibility score with the image, normalizes scores across four candidates, and selects the maximum. TAB-VCR expresses this directly as

$$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$

with an analogous expression for rationales,

$r$0

Training uses cross-entropy for each subtask, and there is no explicit multi-task joint loss beyond reusing the same architecture and encoder weights for both tasks (Lin et al., 2019). KVL-BERT describes the same procedure in Transformer form: a scalar “class score” is produced per candidate, a softmax over the four scores defines a probability distribution, cross-entropy drives the correct candidate upward, and inference sorts candidates by score and chooses the top one (Song et al., 2020).

The benchmark is notable because answers average 7.5 words and rationales average 16 words, making the response space substantially more compositional than standard short-answer VQA (Tang et al., 2021). This affects ranking difficulty: the model must not only detect grounded objects and attributes but also score longer, semantically entangled candidate explanations.

3. Architectural mechanisms for benchmark VCR ranking

A substantial portion of the VCR literature can be read as a sequence of increasingly structured ranking functions.

TAB-VCR begins from the R2C baseline and shows that a much simpler pruned architecture can perform better with fewer trainable parameters. Its joint encoder uses fixed BERT embeddings, a frozen image CNN, token-wise concatenation of visual and linguistic features, a Bi-LSTM, and a small MLP scorer. It also augments region features with attributes and improves text-image grounding by discovering new tags via POS tagging, WordNet first synsets, and Wu-Palmer similarity. The best validation configuration, “Base + ResNet101 + Attributes + New Tags,” reports $r$1 for Q→A, $r$2 for QA→R, and $r$3 for Q→A+R, with approximately $r$4M trainable parameters; on the private test leaderboard, TAB-VCR improves over R2C by $r$5 on Q→A, $r$6 on QA→R, and $r$7 on holistic VCR (Lin et al., 2019).

KVL-BERT moves the ranking function into a single-stream cross-modal Transformer augmented with external commonsense knowledge from ConceptNet 5.5. For each token in the input sentence, it retrieves ConceptNet facts, sorts them by edge weight, and injects the top-$r$8 concepts directly as textual tokens after the source token. Relative position embeddings preserve the original sentence structure, and a visible matrix with mask-self-attention constrains the injected knowledge to interact only with its source token rather than with unrelated tokens or visual features. The paper reports best performance at $r$9 and shows that removing either relative position embedding or mask-self-attention reduces validation performance. KVL-BERT LARGE reaches $q$0 on validation and $q$1 on test for Q→A, QA→R, and Q→AR, respectively (Song et al., 2020).

DMVCR introduces a dynamic working memory intended to store “commonsense between sentences.” After visual grounding, BiLSTM encoding, and cross-modal attentional fusion, the model uses an encoder LSTM to produce a hidden state $q$2 and reads a learned dictionary

$q$3

via content-based addressing:

$q$4

The dictionary is trainable and updated during training; there is no explicit recurrent write gate. The reported benchmark result is $q$5 for Q→A, $q$6 for QA→R, and $q$7 for Q→AR, compared with the R2C baseline at $q$8, $q$9, and $I$0 (Tang et al., 2021).

MEP3P extends candidate ranking with pseudo-3D scene structure and a frozen MLLM. It estimates monocular depth, assigns each detected object a depth coordinate, fuses appearance with pseudo-3D box and depth features, and modifies Transformer attention with a depth-aware bias so that objects at similar depths receive larger attention weights. It also tags answer words with pseudo depth to realize depth-aware association between answer words and objects, and uses BLIP-2 after replacing grounded tags such as “[person 1]” with linguistic labels such as “person.” On the VCR validation set, MEP3P reports $I$1 for Q→A, $I$2 for QA→R, and $I$3 for Q→AR, compared with $I$4, $I$5, and $I$6 without BLIP-2 (Zhu et al., 2023).

VQA-GNN frames candidate ranking as graph reasoning over unstructured and structured multimodal knowledge. It constructs a scene graph from image detections and relations, a ConceptNet-derived concept graph for the question-answer context, and a QA-context super node $I$7 encoded by RoBERTa-large. Two modality-specialized relation-aware GNNs update $I$8 separately from the scene and concept modalities, mitigating the representational gap that appears when visual and symbolic nodes are mixed too early. Candidate scores are then computed from pooled graph features and normalized by softmax. On the VCR test set, VQA-GNN reports $I$9 for Q→A, $C$0 for QA→R, and $C$1 for Q→AR; the hybrid RESERVE-L + VQA-GNN reaches $C$2, $C$3, and $C$4 (Wang et al., 2022).

A separate line addresses ranking under distribution shift rather than raw in-distribution accuracy. “Causal Debiasing for Visual Commonsense Reasoning” analyzes co-occurrence and statistical biases, introduces VCR-OOD-QA and VCR-OOD-VA, and replaces the observational distribution with a backdoor-adjusted interventional distribution by adding an expected confounder embedding $C$5 derived from a dictionary of correct answers or rationales. On VCR-OOD-QA, the method improves MSGT from $C$6 to $C$7; on VCR-OOD-VA, it improves MSGT from $C$8 to $C$9. The same paper reports a small in-distribution trade-off on VCR validation, where MSGT drops from $D$0 to $D$1 (Zou et al., 23 Oct 2025).

Taken together, these systems show that benchmark VCR ranking has been improved through stronger grounding, attribute enrichment, external knowledge injection, dynamic memory, pseudo-3D geometry, multimodal graph reasoning, and causal debiasing. The reported failure cases and OOD trade-offs also indicate that high candidate-ranking accuracy does not eliminate ambiguity in rationale grounding.

4. QuizRank and image-level Visual Concept Ranking

QuizRank uses “Visual Concept Ranking” in a different sense: given a concept $D$2 and a pool of candidate images $D$3, the goal is to rank images by how well they depict the key visual characteristics of $D$4 and, in the contrastive setting, how well they distinguish $D$5 from a distractor concept $D$6 (Ji et al., 18 Sep 2025).

The pipeline converts a concept’s Wikipedia article into multiple-choice questions about important visual characteristics. GPT-4o is the primary LLM for question generation, with LLaMA-4-Scout-17B used for validation; GPT-4o is also the primary VLM for answering, again with LLaMA-4-Scout-17B for replication. The questions are required to be answerable from the image, to avoid leaking concept identity, and to follow accepted MCQ design guidelines. The typical output is 4–11 questions per concept, with mode 5, and each question has four options (Ji et al., 18 Sep 2025).

For each candidate image $D$7, the VLM is prompted to answer every question using only visual evidence. The paper’s base score is the normalized count of correct answers:

$D$8

Images are ranked by $D$9 in descending order. If a distractor image of $a^*$0 performs similarly to a top in-class image of $a^*$1, specifically when the raw-score difference is at most two correct answers, QuizRank triggers a contrastive stage. It then generates contrastive questions from the distinctive visual features of $a^*$2 versus $a^*$3 and re-scores images with

$a^*$4

The paper notes that the final ranking can use either the contrastive score alone or a combination such as lexicographic ordering on $a^*$5 then $a^*$6 (Ji et al., 18 Sep 2025).

The evaluation uses 94 Wikipedia topics from Silva et al. (2024), of which 89 remain after excluding 5 non-visual or too-short pages. Each concept has four images: three in-class images from Wikimedia Commons and one visually similar distractor. The human study uses 320 Prolific participants on a 40-concept subset, with two annotators per image. On that subset, the per-image average accuracy correlation between VLM and humans is Pearson $a^*$7, $a^*$8. The open-source replication reports $a^*$9 between LLaMA rankings and human rankings, with $r^*$0 concepts showing perfect rank correlation and $r^*$1 showing strong or very strong correlation (Ji et al., 18 Sep 2025).

QuizRank also reports separation between in-class and distractor images. For humans, non-distractors have mean $r^*$2 and distractors $r^*$3; for the VLM, non-distractors have $r^*$4 and distractors $r^*$5, with both Kruskal–Wallis and ANOVA yielding $r^*$6. In the 89-concept contrastive pipeline, the base quiz yields in-class $r^*$7 distractor for 57 concepts, 26 ties, and 6 cases where distractor $r^*$8 in-class; for the 51 close cases with $r^*$9, contrastive questions improve separation to 34 in-class wins, 16 ties, and 1 distractor win. Rank stability analysis shows mean Spearman correlation to a 10-question ranking exceeding $a^*$0 from approximately 6 questions upward (Ji et al., 18 Sep 2025).

This formulation recasts ranking as a functional educational test: images are not scored by raw text-image similarity, but by how well they enable correct answers to visual questions derived from an article.

5. Concept-level ranking in interpretable models and model auditing

G-CBM gives perhaps the most literal formalization of Visual Concept Ranking. The model first discovers concept bases by non-negative matrix factorization over patch-level features:

$a^*$1

For a new image, each patch feature is projected onto the frozen concept bases by nonnegative least squares, weak region-level scores are suppressed by a tunable threshold $a^*$2, concept-node features are formed by weighted aggregation of patch embeddings, and a fully connected concept graph is processed by a multi-head GAT (Hossain et al., 1 Jul 2026).

In G-CBM, VCR is the ordering of concept nodes by gradient sensitivity for the predicted class $a^*$3:

$a^*$4

Patch-level grounding uses

$a^*$5

with NNLS coefficients treated as constant during gradient computation. The paper validates the faithfulness of this ranking by deletion and insertion analysis. With ResNet-50 and $a^*$6, the Most-Relevant-First ranking yields AUC$a^*$7/AUC$a^*$8 of $a^*$9 on HAM10000, $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$0 on PH2, $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$1 on Derm7pt, and $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$2 on ImageNet, outperforming random-order controls on all datasets. It also reports an average relative AUC improvement of $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$3 over a ResNet-50 baseline, with peak AUC $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$4 on PH2 using only 2 of 10 concepts and $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$5 on HAM10000 using 3.8 of 9 concepts (Hossain et al., 1 Jul 2026).

The 2026 medical auditing paper uses a different concept-ranking pipeline targeted at large multimodal models. A separate VLM, OpenCLIP, provides concept “labels” by normalized cosine similarity between image and concept text embeddings,

$$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$6

and ridge regression learns concept activation vectors in the target LMM’s residual-stream activation space:

$$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$7

For a fixed completion such as “malignant,” the task score is the length-normalized log-probability of the completion, and concept sensitivity is the average directional derivative

$$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$8

Concepts are ranked by $$p(A_k \mid q, I) = \frac{\exp(s_A(k))}{\sum_{m=1}^4 \exp(s_A(m))},$$9, and significance is assessed by bootstrap resampling with a two-sided one-sample $r$00-test and Bonferroni correction (Janizek et al., 4 Feb 2026).

On a synthetic benchmark with causal ground truth, this medical VCR reports Pearson $r$01, $r$02 between VCR sensitivity and measured interventional effect across 800 conditions. In an adversarial distribution-shift setup, VCR correctly infers the causal sign for the spurious feature $r$03 of the time, while a CLIP-only correlational baseline does so $r$04 of the time (Janizek et al., 4 Feb 2026). On dermatology, top concepts include “tattoo” and “Purpura/Petechiae,” which inspection shows are actually capturing blue or purple dermatology ink markings; manual interventions with four purple or blue dots produce the predicted subgroup-dependent changes in malignancy log-probability. On chest radiographs, “wiring” becomes a strong positive concept for “abnormal,” with top-activating images showing pacemaker wires and EKG leads (Janizek et al., 4 Feb 2026).

The contrast with G-CBM is instructive. G-CBM ranks discovered concept nodes inside a concept bottleneck graph, whereas the medical auditing framework ranks textual concepts projected into LMM activation space. Both, however, treat VCR as a causal-importance ordering rather than as answer selection.

6. Recurrent themes, evaluation criteria, and limitations

Across these disparate formulations, three technical themes recur. First, VCR almost always requires an explicit scoring function over alternatives. In benchmark VCR, the alternatives are answer and rationale candidates scored by MLPs or classifier heads over grounded multimodal representations (Lin et al., 2019, Song et al., 2020). In QuizRank, the alternatives are images scored by per-question correctness (Ji et al., 18 Sep 2025). In G-CBM and medical auditing, the alternatives are concepts scored by gradient sensitivity or directional derivatives (Hossain et al., 1 Jul 2026, Janizek et al., 4 Feb 2026).

Second, high-quality ranking depends on constraining spurious interactions. KVL-BERT uses relative position embeddings and a visible matrix so that injected ConceptNet tokens do not overpropagate through the sequence; the paper reports that $r$05 introduces knowledge noise and slightly degrades accuracy, and that removing mask-self-attention reduces performance (Song et al., 2020). Causal debiasing for benchmark VCR uses backdoor adjustment and a negative-image regularizer because co-occurrence and statistical biases otherwise distort answer and rationale ranking under OOD shifts (Zou et al., 23 Oct 2025). Medical VCR likewise emphasizes robustness to distribution shift and uses manual interventions to test whether ranked concepts reflect actual dependencies rather than correlations in the audit dataset (Janizek et al., 4 Feb 2026).

Third, grounding remains a persistent bottleneck. TAB-VCR improves candidate ranking by augmenting attributes and recovering missed tags, and reports that correct responses more often include new tags than incorrect ones (Lin et al., 2019). DMVCR can retrieve high-level commonsense yet still fail on the precise rationale, including cases where it selects the correct answer but the wrong rationale, or vice versa (Tang et al., 2021). MEP3P shows that depth-aware reasoning helps when 2D proximity is misleading, but it is also sensitive to monocular depth noise and requires replacement of grounded tags with linguistic labels for BLIP-2, which may lose fine-grained grounding information (Zhu et al., 2023).

The literature also contains several domain-specific limitations. QuizRank notes that injection of more or less discriminative questions is constrained by article quality, that domain shift beyond Wikipedia is untested, and that images with occlusions or busy backgrounds can lower discriminability (Ji et al., 18 Sep 2025). G-CBM is sensitive to the threshold $r$06 and the chosen number of concepts $r$07; fixed-size patches may fragment structures, and unsupervised NMF quality can vary by domain and backbone (Hossain et al., 1 Jul 2026). Medical VCR underperforms on purely spatial concepts such as top/bottom and left/right, requires gradient access to the audited model, and can suffer concept-name drift, as when a label such as “Purpura/Petechiae” functions operationally as an ink-marking detector (Janizek et al., 4 Feb 2026).

A common misconception is to treat all “VCR” papers as addressing the same benchmark. QuizRank explicitly states that its “Visual Concept Ranking” is different from the well-known VCR dataset (Ji et al., 18 Sep 2025). Another misconception is to assume that concept ranking always refers to interpretable visual parts. In benchmark VCR, the ranked entities are usually answer and rationale candidates; in G-CBM they are unsupervised concept nodes; in LMM auditing they are textual concept directions; and in QuizRank they are candidate images. This suggests that the acronym now names a broader methodological pattern: multimodal systems that order semantically meaningful alternatives under explicit ranking objectives, but do so at different semantic levels and for different evaluation purposes.