EXAMS-V: Multimodal & Multilingual Exam Benchmark

• EXAMS-V is a benchmark of interleaved exam questions requiring joint OCR, layout parsing, and subject-specific reasoning across multiple languages.
• It comprises 20,932 questions from 20 subjects and 11 languages, sourced from authentic state examinations worldwide.
• Evaluation reveals modality-specific challenges, with models showing varied accuracy on tables, graphs, and symbols in zero-shot settings.

EXAMS-V is a multi-discipline, multilingual, multimodal benchmark built from real school examinations across countries to evaluate vision–LLMs on unified “question snapshot” images that interleave text with figures, tables, maps, scientific symbols, and equations. It extends the earlier EXAMS exam-QA resource into a setting where OCR, layout understanding, and subject-specific reasoning must be performed jointly rather than on separately parsed inputs. The benchmark contains 20,932 multiple-choice questions spanning 20 high-school disciplines, 11 languages, and 7 language families, with 5,086 multimodal items and a train/test split of 16,724 and 4,208 instances (Das et al., 2024).

1. Definition and benchmark design

EXAMS-V was introduced as a realistic evaluation set for multimodal multilingual reasoning by sourcing actual school exam questions from multiple education systems rather than repurposing predominantly English internet data. Its central design choice is interleaving: each sample is a single image crop containing the complete question context, including both textual and visual material. This forces a model to perform OCR, layout parsing, cross-modal grounding, and answer selection in one pipeline, which differs materially from benchmarks that provide an image plus separately extracted text (Das et al., 2024).

The benchmark is restricted to multiple-choice questions with 3–5 options and exactly one correct answer. The subject inventory spans natural sciences, social sciences, and other domains such as business, ethics, fine arts, tourism, agriculture, informatics, and Islamic or religious studies. The resulting tasks combine multilingual reading, region-specific curricular knowledge, and visual reasoning; examples highlighted in the benchmark description include Chinese Gaokao items, English JEE Advanced problems, Arabic questions with nonstandard option formatting, and professional examinations from Poland (Das et al., 2024).

Property	Value
Total questions	20,932
Subjects	20
Languages	11
Language families	7
Multimodal items	5,086
Train / test split	16,724 / 4,208

Compared with M3Exam, EXAMS-V is interleaved rather than non-interleaved, covers more languages, and has greater subject breadth per language, with a minimum of 3 subjects, a maximum of 13, and an average of 7.1 subjects per language. Its multimodal coverage is also larger, with 5,086 multimodal items versus 2,816 in M3Exam (Das et al., 2024).

2. Data sources, curation, and composition

The data are based on official state examinations from ministries of education in multiple countries, inherited in part from EXAMS and extended with English and Chinese material. The English component comes from JEE Advanced in India, and the Chinese component from Gaokao in China. The curation pipeline retraces original exam PDFs, converts them into page images, and manually crops each eligible multiple-choice question as an image containing all associated visual context by using the OpenLabeling tool. Each sample is then assigned JSON metadata comprising a unique identifier, file path, subject, grade, language, and correct answer; annotators additionally mark whether a crop contains visual context or is text-only (Das et al., 2024).

The final release covers grades 4–12. Per-language counts include, for example, 724 English questions, 2,635 Chinese, 439 French, 819 German, 1,645 Italian, 823 Arabic, 2,511 Polish, 3,801 Hungarian, 2,132 Bulgarian, 3,969 Croatian, and 1,434 Serbian questions. The 83 originally collected subjects were aggregated into 20 unified subjects to normalize cross-country curricular naming variation. At the category level, natural sciences account for 53.02% of the benchmark, social sciences for 27.15%, and other subjects for 19.82% (Das et al., 2024).

Cross-lingual parallelism is a notable structural feature. The benchmark includes 1,207 Serbian questions and 1,147 Italian questions with Croatian parallels, as well as 262 Arabic–English parallel questions in science, physics, chemistry, and biology. Balanced sampling by language and subject was used for the train/test split, and parallel questions were kept consistent across splits (Das et al., 2024).

Quality control was conducted by blind checking 50 randomly sampled questions for seven languages against four binary criteria: image clarity, question clarity, single-correct-answer validity, and other issues. Bulgarian, Croatian, Serbian, Italian, and Arabic had all sampled questions pass. Chinese had one unclear image and one unclear question, while English had one invalid sample because the answer appeared in the image (Das et al., 2024).

3. Evaluation protocol and empirical performance

EXAMS-V is evaluated in zero-shot mode without fine-tuning or in-context learning. Models are instructed to return answers in JSON form such as {"answer": "choice"} for automatic extraction, and performance is reported with standard multiple-choice accuracy. Because the questions have 3–5 options, the random baseline varies from 19% to 26% across languages (Das et al., 2024).

The initial benchmark study evaluated native VLMs and OCR-plus-captioning augmented LLMs. Among the VLMs, GPT-4V obtained the highest average score at 42.78%, followed by Gemini-Pro-Vision at 31.13%. Augmented LLM pipelines performed better on average: GPT-4 with OCR and captioning reached 47.11%, Gemini Pro with OCR and captioning 43.99%, and GPT-3.5 Turbo with OCR and captioning 39.47% (Das et al., 2024).

Performance is strongly language- and modality-dependent. Chinese is difficult for all reported systems, with GPT-4V at 22.20%, Gemini-V at 24.56%, and GPT-4 plus OCR and captioning at 23.57%; the benchmark attributes this to heavy visual content and complex reasoning. English is also difficult because the English subset is drawn from JEE Advanced, where GPT-4V scored 29.27%, Gemini-V 29.20%, and augmented GPT-4 32.40%. Arabic introduces format complications because some questions do not label options as A/B/C/D, yielding 24.29% for GPT-4V, 19.38% for Gemini-V, and 26.47% for augmented GPT-4 (Das et al., 2024).

Modality-specific analysis shows that VLM errors are not distributed uniformly across visual types. For GPT-4V, reported accuracies were 52.78% on symbols, 60.00% on figures, 42.00% on graphs, 27.50% on tables, and 62.00% on text. Gemini-V was relatively better on tables at 37.50% but much weaker on symbols, figures, and graphs, with 25.00%, 22.00%, and 26.00%, respectively. The benchmark therefore identifies tabular and graph comprehension, script robustness, and dense exam layouts as primary failure modes rather than treating “multimodality” as a single homogeneous skill (Das et al., 2024).

A common misconception is that passage-through vision alone is sufficient for exam reasoning. The reported results instead show that decoupling perception and reasoning through OCR and captioning can outperform native end-to-end VLMs on this benchmark. Another misconception is that multilingual evaluation is reducible to translation. EXAMS-V explicitly stresses region-specific knowledge and exam formatting conventions, and its Croatian–Serbian–Italian parallels show language-dependent accuracy differences even when content is parallel (Das et al., 2024).

4. Answerability-based formulations and versioned exam banks

A distinct but related line of work uses the term EXAM rather than EXAMS-V and replaces passage-level relevance judgments with answerability of exam questions. In that formulation, EXAMS-V is understood as an “EXAM-style, versioned” setup: a versioned bank of exam questions per topic together with answerability-based scoring. Let $Q=\{q_1,\ldots,q_m\}$ denote the question set for a query, and let a system response be either a ranked list of passages or generated text segmented into passages. A passage is treated as relevant if it contains enough information to answer query-specific exam questions (Farzi et al., 2024).

Two concrete measures are defined. The recall-oriented metric is EXAM Cover,

$$\text{EXAM Cover}(P)=\frac{1}{|Q|}\left|\bigcup_{p\in P}\{q \mid \text{correct}(q,p), \forall q\in Q\}\right|,$$

where answerability can be instantiated either by answer verification against a gold answer or by a self-rating threshold $A_T(q,p)=[\text{rating}(q,p)\ge T]$. The precision-oriented metric is EXAM Qrels, which maps answerability into binary or graded relevance labels per passage and can then be evaluated with standard trec_eval metrics such as Precision@k, MAP, and NDCG (Farzi et al., 2024).

This framework shifts human labor from passage labeling to question-bank curation. Human judges edit and define $Q$, while an LLM-based grader determines which questions are answerable from which passages. The paper emphasizes several operational consequences: post-hoc expansion of the question bank, ongoing evaluation of future retrieval or generation systems, and elimination of “holes” caused by unjudged documents because new passages can always be graded automatically (Farzi et al., 2024).

Empirically, the answerability paradigm aligns closely with established leaderboards. On TREC CAR Y3 with TQA questions, EXAM Cover correlates with the official leaderboard at Spearman $\approx 0.94$ and Kendall $\tau \approx 0.84$; on TREC DL 2019 and 2020, EXAM Qrels reaches Spearman $\approx 0.93$ and $\approx 0.96$, respectively. Passage-level inter-annotator agreement can remain low, such as Cohen’s $\kappa \approx 0.08$ on CAR Y3 with TQA, while leaderboard correlations remain high. The point of the method is thus system ranking rather than faithful reconstruction of passage-level qrels (Farzi et al., 2024).

Versioning is explicit in this line of work: exam banks are denoted $Q^v$, maintained with changelogs, and regraded to produce updated $$\text{EXAM Cover}(P)=\frac{1}{|Q|}\left|\bigcup_{p\in P}\{q \mid \text{correct}(q,p), \forall q\in Q\}\right|,$$0. That interpretation gives EXAMS-V a second meaning beyond the multimodal benchmark: a version-controlled, human-curated exam bank for longitudinal evaluation (Farzi et al., 2024).

5. Localized and application-specific EXAMS-V-style resources

Subsequent work has produced localized resources that function as EXAMS-V-style benchmarks even when the exact name is not reused. ViExam is a Vietnamese multimodal examination benchmark containing 2,548 multiple-choice questions across Mathematics, Physics, Chemistry, Biology, Geography, Driving Test, and IQ Test. It is explicitly positioned as a Vietnamese multimodal extension to the exam-benchmark ecosystem. On ViExam, state-of-the-art VLMs achieve 57.74% mean accuracy, open-source models 27.70%, average human test-takers 66.54%, human best 99.60%, and the strongest reported model, o3, 74.07%. Cross-lingual prompting with English instructions while keeping Vietnamese content lowers accuracy by about 1 percentage point for the strongest models, whereas human-in-the-loop collaboration improves performance by about 5 percentage points (Dang et al., 19 Aug 2025).

V-Math, an agentic framework for the Vietnamese National High School Graduation Mathematics Exams, does not introduce an explicit EXAMS-V benchmark, but it creates two resources that functionally serve EXAMS-V-style evaluation in Vietnamese: a 500-set NHSGMEs dataset aligned to the newest 2025+ MOET blueprint and a public VNHSGE set of 250 official questions from 2019–2023. Its evaluation reports exam-level, section-wise, item-level, and agent-wise metrics. On VNHSGE, V-Math reports 100% exact-match accuracy every year; on 50 held-out NHSGME exams it reports section-wise accuracies of 98.1% for Part I, 93.8% for Part II, and 88.4% for Part III, with full-exam set-level accuracy of 64% and item accuracy of 92.1% (Nguyen et al., 12 Sep 2025).

A Spanish-language variant of the exam-benchmark idea appears in the evaluation of GPT-3.5 and Bard/Gemini Pro on official undergraduate admissions exams from Mexico’s Instituto Politécnico Nacional. The curated IPN exam set comprises Spanish multiple-choice questions, plus an English reading-comprehension section, across Engineering/Mathematical and Physical Sciences, Biological and Medical Sciences, and Social and Administrative Sciences. GPT-3.5 achieved 60.94% overall and Bard 60.42%, with GPT-3.5 stronger in Mathematics and Physics and Bard stronger in History and other factual tasks. The study argues that the dataset could serve as a Spanish variant or component of an EXAMS-style benchmark because it combines multi-subject coverage, official acceptance thresholds, and authentic admissions difficulty (Miranda et al., 2023).

The educational significance of authentic exams has also been reinforced by direct model-taking studies. In a remote Software Engineering BSc context, GPT-4 scored 35/38 points, corresponding to grade A, on a Reactive Programming open-book exam and 40/68 points, corresponding to grade D, on an Artificial Intelligence open-book exam. That work concludes that the examined open-book exams were not GPT-4-proof and recommends oral or redesigned hybrid formats where necessary (Dobslaw et al., 2023). A plausible implication is that authentic exam artifacts now serve simultaneously as assessment instruments and as model stress tests.

6. Limitations, disambiguation, and future directions

EXAMS-V inherits several limitations from its construction choices. It includes only multiple-choice questions and does not provide rationales or step-by-step explanations. Its modality analysis is limited to broad categories such as symbols, figures, graphs, and tables, and cross-country difficulty variation complicates strict comparability across languages. Parallel subsets partially address this issue, but they currently exist mainly for Croatian–Serbian–Italian and a smaller Arabic–English subset (Das et al., 2024).

The answerability-based EXAM formulation has its own failure modes. LLM graders may hallucinate or mis-rate passages, automatically generated questions may miss critical topical facets, and cost and latency rise when grading large passage pools. The framework is therefore strongest when human curation remains in the loop, especially for question-bank definition, threshold selection, and post-hoc refinement (Farzi et al., 2024).

The term is also susceptible to acronymic confusion. A separate line of work uses EXAMS to denote the “EXtensible Animator for Mobile Simulations,” an NS-2 visualization tool for mobile wireless traces whose architecture separates a general animation engine from protocol-specific Java modules loaded by reflection (0902.4527). Other exam-related systems focus not on benchmark design but on administration and integrity enforcement: one visual-analytics approach combines webcam video, mouse events, head pose, Blur/Focus, and Copy/Paste signals for online-exam proctoring, while another video-only pipeline detects another person, electronic devices, and candidate absence from webcam footage (Li et al., 2021, Özgen et al., 2021). These systems address remote examination operations rather than multilingual multimodal question answering.

Future work in the EXAMS-V benchmark tradition is already sketched in the literature. Proposed directions include expanding subjects, languages, image samples, and modality taxonomies; maintaining versioned exam banks $$\text{EXAM Cover}(P)=\frac{1}{|Q|}\left|\bigcup_{p\in P}\{q \mid \text{correct}(q,p), \forall q\in Q\}\right|,$$1 with changelogs; generating domain-specific or multilingual question banks; extending evaluation to image, video, and audio passages; and integrating better answer verification, human-in-the-loop review, and calibration studies for self-rating reliability (Das et al., 2024, Farzi et al., 2024). Collectively, these strands position EXAMS-V not as a single fixed dataset alone, but as a broader paradigm for evaluating models against authentic, evolving examinations under multilingual and multimodal constraints.