Contrastive QuizRank

• The paper introduces Contrastive QuizRank, extending QuizRank by explicitly modeling feature differences between target and distractor concepts.
• It computes contrastive feature differences using CLIP embeddings and generates targeted multiple-choice questions that assess image distinctiveness.
• Experimental results demonstrate improved discriminative ranking in fine-grained recognition settings, aligning closely with human evaluations.

Searching arXiv for the specified paper and nearby relevant context. Contrastive QuizRank is an image-ranking method introduced within QuizRank for selecting images as learning interventions by quizzing vision–LLMs (VLMs) on multiple-choice questions derived from textual descriptions of a target concept. In the QuizRank framework, images are ranked according to how well they enable a VLM to answer questions about important visual characteristics of the concept. Contrastive QuizRank extends this procedure to cases involving visually similar but semantically distinct concepts by explicitly modeling feature differences between a target concept and one or more distractor concepts, then generating questions that emphasize those distinguishing attributes. The method is presented in “QuizRank: Picking Images by Quizzing VLMs” (Ji et al., 18 Sep 2025).

1. Position within QuizRank

QuizRank is motivated by the observation that images improve the readability and comprehension of Wikipedia articles by serving as “illustrative aids,” while image selection remains uneven because not all images are equally effective and not all editors are trained in their selection (Ji et al., 18 Sep 2025). The base QuizRank procedure transforms textual descriptions of an article’s subject into multiple-choice questions about important visual characteristics, then uses a VLM to answer those questions for each candidate image. Images that better support correct answers receive higher ranks.

Standard QuizRank generates multiple-choice questions solely from the target article’s visual description. This is reported to be effective at identifying images that exhibit key in-class features, but it can fail to discriminate between visually similar yet semantically distinct concepts such as “Western Bluebird” and “Mountain Bluebird” (Ji et al., 18 Sep 2025). Contrastive QuizRank is introduced specifically to address this failure mode.

The defining premise of Contrastive QuizRank is that question generation should focus not only on what characterizes the target concept in isolation, but also on what differentiates it from plausible distractors. The method therefore augments the base pipeline by explicitly modeling the feature differences between a target concept $C_t$ and one or more distractor concepts $\{C_d\}$. This suggests a shift from generic visual adequacy to discriminative pedagogical adequacy: an image is valuable not merely because it depicts the target, but because it supports identification of the target against nearby alternatives.

2. Formalization and contrastive objective

The paper defines the target concept as $C_t$, the distractor set as $\{C_d\}$, and the pool of candidate images as

$I = \{I_1, I_2, \ldots, I_n\}.$

It further defines $f(\cdot)$ as a feature extractor, for example a CLIP embedding of text description or canonical image features (Ji et al., 18 Sep 2025).

For each distractor $C_d$, Contrastive QuizRank computes a contrastive feature difference

$$\Delta f_{t,d} = f(T_{ext}(C_t)) - f(T_{ext}(C_d)),$$

where $T_{ext}(\cdot)$ denotes the textual description of the concept. Intuitively, $\Delta f_{t,d}$ highlights features more prominent in $\{C_d\}$0 than in $\{C_d\}$1 (Ji et al., 18 Sep 2025). The method uses these feature differences to drive question generation. Concretely, if feature dimension $\{C_d\}$2 has large positive $\{C_d\}$3, the system solicits a multiple-choice question about that attribute.

At a high level, the method extracts visual feature vectors for $\{C_d\}$4 and each $\{C_d\}$5, computes their difference $\{C_d\}$6, translates $\{C_d\}$7 into multiple-choice questions that ask “Which feature in this image differentiates $\{C_d\}$8 from $\{C_d\}$9?”, then quizzes each candidate image. Images of $C_t$0 are expected to answer contrastive questions correctly, while $C_t$1 images are expected to fail (Ji et al., 18 Sep 2025).

This contrastive construction narrows the evaluation target. Rather than rewarding images simply for exposing canonical features of the target, it rewards images whose visual evidence distinguishes the target from confusable neighbors. A plausible implication is that the ranking criterion becomes especially relevant in fine-grained recognition settings, where within-category similarity is high and coarse text–image alignment may be insufficient.

3. Contrastive question generation

The question-generation algorithm takes as input the text of the target concept $C_t$2, the texts of distractors $C_t$3, and an optional image subschema such as Appearance or Design sections; it outputs a contrastive multiple-choice question set $C_t$4 (Ji et al., 18 Sep 2025). The procedure first computes embeddings $C_t$5 and $C_t$6 for each distractor and forms the set of differences $C_t$7.

For each contrastive feature difference, the algorithm selects the top-$C_t$8 dimensions with highest positive weight in $C_t$9, then translates each selected dimension into a textual stem via an LLM prompt that describes the attribute without naming the concept. The prompt asks the model to identify a visual feature that distinguishes one description from another and then craft a generic question such as “What feature is depicted in the image?” with one correct answer from the target’s unique feature set and three plausible distractors drawn from the distractor’s unique features (Ji et al., 18 Sep 2025). For each stem, the LLM generates a question stem, a correct answer, and distractors.

The structured description given for this stage includes three explicit constraints: identify unique visual characteristics of $\{C_d\}$0 versus $\{C_d\}$1, prompt the LLM for contrastive question generation, and enforce multiple-choice design best practices, namely one correct answer, three plausible distractors, no concept names, and no “leaked” facts (Ji et al., 18 Sep 2025). These constraints are intended to ensure that the VLM must attend to the image rather than exploit textual shortcuts.

Implementation details specify OpenAI GPT-4o as the LLM for question generation, also validated with LLaMA-4-Scout-17B, using chain-of-thought prompting to list distinct and overlapping features and then generate multiple-choice questions (Ji et al., 18 Sep 2025). Questions are returned in JSON arrays of objects containing "question", "options", "correct_answer", and "rationale". The paper reports that the number of questions per concept varied from 4 to 11, with mode 5. Contrastive multiple-choice questions are only generated when a baseline quiz fails to strongly separate distractors, operationalized as score difference $\{C_d\}$2 (Ji et al., 18 Sep 2025).

The use of contrastive prompting in this stage is central. The generated questions are generic in wording, but their semantics are anchored in differential attributes between target and distractor descriptions. This suggests that the quality of the contrastive pipeline depends not only on the VLM’s visual competence but also on the LLM’s ability to convert latent feature differences into well-formed, non-leaky assessment items.

4. VLM quizzing and scoring

For each image $\{C_d\}$3 and question $\{C_d\}$4, the pair $\{C_d\}$5 is submitted to the vision–LLM. The VLM produces an answer $\{C_d\}$6 and an answer confidence or confidence-based correctness indicator $\{C_d\}$7 (Ji et al., 18 Sep 2025). The per-image, per-question score is defined as

$\{C_d\}$8

The overall image-ranking score is then

$\{C_d\}$9

The paper also notes that the score may be normalized by $I = \{I_1, I_2, \ldots, I_n\}.$0 or weighted:

$I = \{I_1, I_2, \ldots, I_n\}.$1

to emphasize harder or contrastive questions (Ji et al., 18 Sep 2025).

The VLM used for quizzing is GPT-4o multimodal vision model, with LLaMA-Scout as an open-source alternative (Ji et al., 18 Sep 2025). The pipeline therefore separates question generation from question answering: an LLM constructs assessment items, and a VLM is evaluated on those items using candidate images as evidence.

Within the contrastive setting, the score acquires a discriminative interpretation. An image of the target concept should accumulate correct answers on questions derived from the target’s unique $I = \{I_1, I_2, \ldots, I_n\}.$2-features, whereas distractor images should not. This makes the aggregate score functionally analogous to a test of visual distinctiveness. The paper’s description that “Images of $I = \{I_1, I_2, \ldots, I_n\}.$3 will answer contrastive questions correctly, while $I = \{I_1, I_2, \ldots, I_n\}.$4 images will fail” formalizes the intended separation criterion (Ji et al., 18 Sep 2025).

5. Triggering conditions, distractor selection, and workflow

Contrastive QuizRank is not described as a universal replacement for the baseline pipeline. Rather, it is activated selectively when the baseline quiz does not strongly separate the target from distractors. The reported heuristic is a score difference threshold of less than 2 (Ji et al., 18 Sep 2025). Only in these cases are contrastive multiple-choice questions generated.

Distractor selection uses linked Wikipedia articles or same-category peers (Ji et al., 18 Sep 2025). This means that the contrastive procedure is grounded in nearby concepts that are likely to produce realistic confusion. The paper also states that only if distractor images score similarly on the base quiz does the system trigger the contrastive pipeline.

The resulting workflow can be summarized as a staged ranking process:

Stage	Operation	Condition
Base QuizRank	Generate MC questions from target description	Default
Borderline detection	Check whether baseline quiz strongly separates distractors	Trigger when score difference $I = \{I_1, I_2, \ldots, I_n\}.$5
Contrastive refinement	Generate contrastive MCQs from target–distractor feature differences	Applied only to borderline cases

This staged design indicates that Contrastive QuizRank functions as a refinement mechanism rather than a first-pass scorer. A plausible implication is that the method trades additional inference and prompting cost for improved discrimination precisely where the base system is uncertain or insufficiently selective.

6. Experimental findings

The reported evaluation uses 89 Wikipedia pages from the Silva et al. benchmark, each with 3 in-class images and 1 curated distractor from the same category (Ji et al., 18 Sep 2025). Human evaluation involved 320 Prolific participants over 40 concepts, with 2 labels per image and compensation of $I = \{I_1, I_2, \ldots, I_n\}.$61.25. The models used were GPT-4o and LLaMA-4-Scout-17B as a secondary check (Ji et al., 18 Sep 2025).

Several quantitative findings are reported. First, VLM–human correspondence reached per-image accuracy Pearson $I = \{I_1, I_2, \ldots, I_n\}.$7 with $I = \{I_1, I_2, \ldots, I_n\}.$8 (Ji et al., 18 Sep 2025). The VLM generally outperformed humans on non-distractor images, with mean 0.66 versus 0.44. Second, baseline QuizRank correctly separated in-class versus distractor in 57/89 concepts, using difference $I = \{I_1, I_2, \ldots, I_n\}.$9 as the criterion. With contrastive refinement on 51 borderline cases, separation improved by 34 additional correct (Ji et al., 18 Sep 2025).

Additional checks addressed memorization, popularity, and quiz length. The memorization check found no significant difference in VLM performance on images uploaded before versus after the VLM training cutoff. Popularity versus quality exhibited weak correlation, approximately $f(\cdot)$0, between image usage count and standardized VLM score on 260 images across 13 random concepts (Ji et al., 18 Sep 2025). Quiz size stability showed that rank ordering stabilizes by approximately 6 questions, with Spearman $f(\cdot)$1 relative to a 10-question baseline (Ji et al., 18 Sep 2025).

The reported result profile is consistent with the method’s intended role. The gain on borderline cases is specifically aligned with the stated motivation for contrastive refinement: standard QuizRank can identify images with key in-class features, but may not distinguish them sharply from close distractors. The contrastive mechanism appears designed to convert those ambiguous cases into questions about differentiating attributes, thereby increasing discriminative separation.

7. Strengths, limitations, and prospective extensions

The paper identifies several strengths of the approach. It directly evaluates images as learning interventions through functional question-answering performance; contrastive questions focus the VLM on distinguishing features and improve robustness against semantically similar distractor images; and the system shows high alignment with human judgments while providing a scalable alternative to crowd annotation (Ji et al., 18 Sep 2025).

The limitations are also explicit. The method relies on the quality of article text: short or poorly written descriptions yield fewer or lower-quality visual questions. Abstract or non-visual concepts, including biographical articles, fall outside the scope. VLM reasoning remains susceptible to subtle prompt “leaks,” although the question-design procedure attempts to mitigate this. The current contrastive distractor selection heuristic, score gap $f(\cdot)$2, may miss some cases; future work could automate distractor mining via semantic similarity (Ji et al., 18 Sep 2025).

Several future extensions are proposed: Multilingual QuizRank, adaptation of prompts and VLMs to non-English descriptions, incorporation of question difficulty calibration and dynamic weighting of $f(\cdot)$3, integration of aesthetic heuristics such as composition and blur alongside QA-based ranking, and development of an interactive editor tool for human-in-the-loop question refinement and image authoring suggestions (Ji et al., 18 Sep 2025).

A common misconception would be to treat Contrastive QuizRank as a generic image-quality metric. The description does not support that interpretation. Its operational target is not unrestricted visual quality, but image usefulness as a learning intervention under contrastive questioning. The reported weak correlation between image usage count and standardized VLM score further suggests that conventional popularity signals are not equivalent to the method’s notion of quality (Ji et al., 18 Sep 2025). By grounding ranking in contrastive visual assessment, the method moves beyond coarse text–image alignment toward a more discriminative and pedagogically oriented criterion.