QuizRank: Quiz-Based Wikipedia Image Ranking
- QuizRank is a method that measures an image’s instructional utility by converting text into quiz questions assessing visual properties.
- It uses vision-language models to answer generated multiple-choice questions, ensuring images clearly depict key visual features.
- Contrastive QuizRank refines rankings by comparing distractors to highlight discriminative features for fine-grained visual distinctions.
QuizRank is a method for ranking Wikipedia images by treating image selection as a quiz-based evaluation problem. It converts article text into multiple-choice questions about visually observable properties of the concept, asks a vision-LLM to answer those questions from each candidate image, and ranks images by how well they support correct answers. In this formulation, an image is not evaluated primarily by caption similarity or generic semantic relatedness, but by whether it functions as an effective “illustrative aid” for learning and discrimination; the method also includes a contrastive extension for visually similar concepts (Ji et al., 18 Sep 2025).
1. Concept and problem setting
QuizRank was introduced for Wikipedia image selection, where the underlying editorial objective is not merely to attach any relevant picture to an article, but to choose an image that “look[s] like what [it is] meant to illustrate” and serves as an “illustrative aid.” The method addresses the fact that Wikipedia operates at a scale of millions of articles and a large image pool, while many editors are not trained in image selection. In that setting, images may be generic, ambiguous, or only weakly aligned with the article’s instructional role (Ji et al., 18 Sep 2025).
The central reframing is to treat image choice as an educational-assessment problem. Rather than asking whether an image matches text, QuizRank asks whether the image helps answer questions derived from the text. The paper positions this as a measure of instructional utility: a good image should support “in-class representation,” meaning that it visibly instantiates key features of the target concept, and “between-class representation,” meaning that it helps distinguish the target from similar concepts. This emphasis is particularly important for encyclopedia use, where readers often rely on images not only to recognize an object class but also to disambiguate closely related items (Ji et al., 18 Sep 2025).
This framing also narrows the notion of visual relevance. The best image in QuizRank is not necessarily the most aesthetically pleasing or the most frequently reused across Wikipedia, but the one that best supports question answering about important visual characteristics of the concept. A plausible implication is that QuizRank defines image quality operationally through task performance rather than through editorial popularity or surface text-image alignment.
2. Base pipeline and scoring rule
The base QuizRank pipeline has four conceptual stages. First, it takes Wikipedia article text and uses a LLM to generate multiple-choice questions about visually observable properties. Second, for each candidate image, a vision-LLM answers those questions using only the image as evidence. Third, the image is scored by how many questions the model answers correctly. Fourth, images are ranked by that score (Ji et al., 18 Sep 2025).
The question-generation prompt is explicitly constrained to focus on “visually identifiable features only.” It asks the model to reason about what can be visually inferred from the concept and to turn that reasoning into multiple-choice questions about properties such as shape, texture, material, structure, or other visible attributes, while avoiding non-visual facts such as location or measurements. It also tries to avoid leaking the concept name into the question. The appendix prompt requests JSON output containing the question, four options, the correct answer, and a rationale. The reported quizzes ranged from 4 to 11 questions per concept, with a mode of 5; the method does not force a fixed quiz length because the available visual description depends on the article (Ji et al., 18 Sep 2025).
The answering stage is similarly constrained. The vision-LLM is instructed to answer based only on visible information in the image, not to use prior knowledge or indexed knowledge, and it is allowed to respond that it cannot answer from the image. The paper describes this as important for avoiding contamination from world knowledge. It also reports a memorization or indexing control comparing older Commons images with newer images uploaded after the model’s training cutoff, and found no significant performance difference, which the authors use as indirect evidence against simple dataset memorization (Ji et al., 18 Sep 2025).
The baseline scoring rule is intentionally simple: the image’s score is the normalized number of correct answers. The paper notes that an image-by-question score matrix can also be retained to identify bundles of images when no single image answers everything. It does not commit to a more elaborate main scoring function, although it mentions possible future variants such as weighting questions by difficulty or importance. Separate from QuizRank scoring itself, the paper uses as the popularity measure in a comparison study and standardizes quiz performance with z-scores within each concept (Ji et al., 18 Sep 2025).
3. Contrastive QuizRank
The main refinement is Contrastive QuizRank, which is designed for cases where a correct image and an incorrect but visually similar distractor are difficult to separate. The motivation is that a good encyclopedia image should not only depict the target class well, but should also help distinguish the target from similar alternatives. The contrastive method therefore uses descriptions of both a target concept and one or more distractor concepts, asks the LLM to identify features unique to each and features they share, and then generates questions whose correct answers are specific to the target’s unique features (Ji et al., 18 Sep 2025).
In the appendix prompt, the model is asked to produce three sets of visual features: those distinct to Object A, those distinct to Object B, and shared features. The generated multiple-choice questions are then structured so that the correct answer applies only to Object A, while distractors correspond to features applicable only to Object B. This produces questions that are intended to be more discriminative than the baseline visually grounded questions, particularly in fine-grained recognition settings (Ji et al., 18 Sep 2025).
Contrastive refinement is not applied indiscriminately. The paper first tests the distractor image on the base questions. If the out-of-class image performs nearly as well as the in-class image, specifically when the score difference is less than 2, it triggers contrastive question generation and rescoring. This conditional strategy limits additional computation to cases where the baseline quiz does not sufficiently discriminate (Ji et al., 18 Sep 2025).
The Gujia–Chandrakala case study illustrates the mechanism. With the original questions, Gujia scored 3/5 and Chandrakala 2/5, a weak separation because both sweets shared properties such as being golden-brown and pastry-like. After contrastive question generation, the questions targeted features unique to Gujia, and the resulting split became Gujia 4/4 and Chandrakala 0/4. In the paper’s interpretation, the half-moon, folded, empanada-like shape and single-fold structure were discriminative features that the contrastive formulation successfully surfaced (Ji et al., 18 Sep 2025).
4. Evaluation design and empirical findings
The evaluation uses the benchmark from Silva et al. 2024. The study started from 94 concepts and excluded 5 non-visual or too-short pages, leaving 89 concepts. For each concept, the authors curated four images: three correct in-class images from Wikimedia Commons and one distractor image of a visually similar but incorrect object. They also ensured that the distractor image had been used on some Wikipedia page in any language, so that it would not be an obviously poor candidate (Ji et al., 18 Sep 2025).
Human evaluation was conducted on a subset of 40 concepts. The study recruited 320 Prolific participants, each paid about $1–$1.25 via Qualtrics; each participant saw five concepts, and each image was intended to be rated by two participants. Human participants answered the QuizRank-generated questions after seeing one image, mirroring the model evaluation setup. The main empirical question was whether images that help the vision-LLM also help humans (Ji et al., 18 Sep 2025).
The principal result is a strong correlation between human and model image-level scores. Across the 40-concept subset, the per-image average accuracy correlation between humans and GPT-4o was Pearson with . The paper also reports strong alignment with LLaMA-4-Scout-17B: the Pearson correlation between LLaMA’s image rankings and human judgments was with . In model-to-model comparison, 32 of 40 concepts, or 80%, showed strong or very strong agreement with GPT-4o, and 26 concepts, or 65%, had perfect rank correlation (Ji et al., 18 Sep 2025).
The quizzes also separate correct images from distractors in statistically significant ways. Humans scored higher on non-distractor images than on distractors, with means versus ; the vision-LLM showed the same pattern with versus . Both Kruskal–Wallis and ANOVA tests were significant in each case: for humans, $1–$0 and $1–$1; for the model, $1–$2 and $1–$3. On non-distractor images, the model significantly outperformed humans, while on distractor images the difference was small and not significant (Ji et al., 18 Sep 2025).
A separate comparison examined popularity. On 13 concepts, the authors sampled 20 images per concept and compared image usage frequency across Wikipedia pages to QuizRank performance. They found only a very weak positive trend, indicating that reuse frequency is only weakly related to quiz-based instructional value. The paper is careful to note that popularity may reflect other editorial goals, such as composition or aesthetics, rather than learning utility (Ji et al., 18 Sep 2025).
The contrastive pipeline also improved discrimination. Across all 89 concepts, using only the base questions, the correct image beat the distractor in 57 cases, tied in 26, and lost in 6. In 51 concepts, the gap was small enough to warrant contrastive refinement. After applying contrastive question generation to those 51 cases, the correct image won in 34 cases, tied in 16, and lost only once. This result is the paper’s strongest evidence that the contrastive extension addresses precisely the failure mode it was designed for (Ji et al., 18 Sep 2025).
5. Scope, limitations, and failure modes
QuizRank works best for visually concrete topics. The paper states that the method is less suitable for abstract, historical, or biographical subjects where no meaningful visual representation exists. It also depends on converting article content into strong visually grounded multiple-choice questions, so sparse or poorly written articles may yield weak quizzes and therefore weak rankings (Ji et al., 18 Sep 2025).
The paper further notes that question quality is heavily prompt-dependent and article-dependent. Although the prompts attempt to restrict questions to visible features, the overall method still relies on the LLM’s ability to produce high-quality MCQs. A plausible implication is that errors in question generation can propagate directly into ranking quality, because the ranking function is defined in terms of quiz performance rather than independent visual criteria.
The authors also caution that they cannot prove the vision-LLM never uses latent knowledge. They provide indirect controls, including the old-versus-new image comparison and strict prompting to require image-only reasoning, but they do not claim a formal guarantee against prior-knowledge leakage. More generally, the paper frames QuizRank as a complement to human curation rather than a replacement for editorial judgment, since some image choices on Wikipedia may prioritize neutrality, aesthetics, or broader editorial convention rather than the specific learning utility measured by the quiz (Ji et al., 18 Sep 2025).
Finally, no single image may answer all important questions for some concepts. The paper therefore notes that bundles of images or weighted question sets may be preferable in such cases. This suggests that QuizRank’s current single-image ranking formulation captures one important editorial criterion, but not necessarily the entirety of visual documentation quality.
6. Position within quiz-based AI systems
QuizRank belongs to a broader family of systems in which quizzes are used as operational interfaces for retrieval, navigation, or evaluation rather than only as assessments. In educational video interfaces, QuizCram uses quizzes to help users determine what they should view and prioritizes review with a mastery score based on past performance, fraction of the associated video segment watched, and recency of review (Kovacs et al., 2021). In educational recommendation, NERE formulates session-based recommendation as prediction of a 128-dimensional embedding for the next study set and retrieves candidates by nearest-neighbor search over set embeddings (Nadeem et al., 2018). In authoring workflows, the self-hosted L2Q pipeline converts lecture PDFs into multiple-choice questions using a local LLM plus deterministic quality control, then exports static quiz banks for deployment without runtime LLM calls (Shintani, 20 Feb 2026).
These systems differ in object of ranking. QuizCram ranks lecture segments for review, NERE ranks study items for recommendation, and L2Q validates and exports question banks. QuizRank instead ranks images by asking whether they help a model answer conceptually targeted visual questions. This suggests that the distinctive contribution of QuizRank is not quiz generation alone, but the use of quiz performance as an interpretable proxy for the utility of a non-text artifact—in this case, a Wikipedia image.
The paper also proposes broader applications beyond Wikipedia, including textbooks, dictionaries, guidebooks, and educational articles where images should be selected for instructional value rather than mere appearance. In that broader interpretation, QuizRank is a scalable evaluation method for visual learning aids: by asking a model to answer quiz questions from an image, it estimates whether the image supports the intended concept and whether it helps distinguish that concept from nearby alternatives (Ji et al., 18 Sep 2025).