MISS-QA: Multimodal Diagram Benchmark
- MISS-QA is a benchmark for multimodal information-seeking over scientific papers, featuring 1,500 expert-annotated QA examples from 465 papers.
- It uniquely leverages schematic overview diagrams with colored bounding boxes, forcing models to align visual elements with full paper context.
- Evaluations of 18 multimodal models reveal challenges in calibration and cross-modal reasoning, with human experts outperforming current systems.
Searching arXiv for MISS-QA and a few related scientific-paper QA benchmarks to ground the article with current paper identifiers. {"query":"arXiv (Zhao et al., 14 Jul 2025) MISS-QA schematic diagrams scientific papers", "max_results": 5} {"query":"QASPER question answering scientific papers arXiv", "max_results": 10} {"query":"QASA scientific papers question answering arXiv", "max_results": 10} MISS-QA is a benchmark for Multimodal Information-Seeking over Scientific papers that evaluates whether multimodal foundation models can interpret schematic diagrams in research papers and use them together with paper text to answer realistic, free-form, information-seeking questions. It was introduced as the first benchmark specifically targeting this capability, with 1,500 expert-annotated examples drawn from 465 scientific papers and an evaluation suite spanning 18 frontier multimodal foundation models plus human experts. The benchmark is motivated by a gap between text-only scientific-paper QA and multimodal scientific QA settings that focus mainly on charts, tables, or experimental-result figures, rather than on overview diagrams that summarize methods, workflows, modules, and conceptual relations (Zhao et al., 14 Jul 2025).
1. Conceptual scope and distinctiveness
MISS-QA is defined around a specific document modality: the schematic overview diagram that typically appears as the first or second figure of a scientific paper. In the benchmark, these figures are chosen because they usually present a research overview, system pipeline, architecture, or methodological framework. The task is not figure captioning, OCR, or generic visual question answering in isolation. Instead, a model must use the diagram to identify the referenced component and then consult the broader paper context to answer a question whose resolution often depends on motivations, implementation details, literature grounding, experimental implications, or limitations (Zhao et al., 14 Jul 2025).
The paper positions MISS-QA against two established lines of work. On one side are scientific-paper QA benchmarks such as QASPER and QASA, which are described as essentially text-only. On the other side are multimodal scientific QA or figure benchmarks such as ArXivQA, MMSci, CharXiv, and M3SciQA, which are described as focusing mostly on charts/tables or experimental-result figures. MISS-QA isolates a different capability: understanding diagrams that summarize a paper’s method and then aligning those diagram elements with the full document.
A central design choice is the use of colored bounding boxes. Each question refers to one or two highlighted visual elements using placeholders such as “the module highlighted by the red bounding box.” This prevents a shortcut in which a model answers from the wording of the question alone. Rather than naming a component directly, the question forces diagram inspection first and only then paper-context retrieval.
Two misconceptions are explicitly addressed by the benchmark design. First, MISS-QA is not a pure vision benchmark: the answer is often not visually explicit in the figure alone. Second, it is not reducible to long-context text QA: the question formulation is intended to require identification of the relevant diagram component before the text can be used.
2. Task formulation and question taxonomy
Formally, the task takes a schematic diagram diag, a question q, and the main sections of the paper p, and requires a model to produce a free-form answer a. In plain terms, the model conditions jointly on the visual overview figure and the paper text and generates the most probable answer. The paper describes this as a generative information-seeking task rather than a classification problem.
The benchmark organizes questions into five information-seeking subsets. These subsets are intended to reflect how researchers use overview diagrams as entry points into papers.
| Subset | Focus |
|---|---|
| Design rationale | Motivation or conceptual reason for a highlighted design choice |
| Implementation details | Technical interaction between highlighted modules or procedures |
| Literature background | Relation to prior work or methodological traditions |
| Experimental results | What results imply about a highlighted module’s role or effect |
| Others | Limitations, alternatives, ethics, or related aspects not covered above |
These categories are not merely topical labels. They encode different reasoning demands. Design rationale requires recovering why a module exists; implementation details requires technical interaction analysis; literature background requires placing a module in methodological lineage; experimental results requires connecting the highlighted element to empirical findings; and Others captures residual but still paper-grounded questions.
The benchmark also includes unanswerable questions as an explicit design axis. This is not treated as a separate information-seeking category in the original taxonomy; rather, answerability is orthogonal to the five subsets. A model is therefore expected not only to answer grounded questions but also to abstain and say “I do not know” when the paper does not contain the needed information. This makes calibration and abstention part of the benchmark’s intended scope.
3. Corpus construction and expert annotation
The data construction pipeline is structured around three stated desiderata: the benchmark should reflect real-world research scenarios, require actual schematic diagram interpretation, and cover diverse reasoning types. Source papers were collected from arXiv in AI-related categories: Artificial Intelligence, Computation and Language, Machine Learning, Computer Vision and Pattern Recognition, and Information Retrieval. The paper selection window was July 1 to November 30, 2024, chosen to be later than the cutoff dates for most open-source model pretraining corpora in order to reduce contamination concerns (Zhao et al., 14 Jul 2025).
Filtering was manual and expert-driven. Five expert annotators, one per subfield, excluded non-research papers such as surveys or dissertations, papers whose first two figures were unsuitable schematic diagrams, and papers judged low quality or affected by HTML parsing errors. After filtering, the benchmark retained 465 scientific papers. For each paper, annotators labeled the first figure identified as a schematic diagram, and that figure became the basis for QA annotation.
The annotation workforce consisted of 16 experts in AI-related fields, including research scientists, postdocs, and PhD students across ML, CV, NLP, IR/NLP, CV/NLP, and DM. Every annotator had authored at least three AI-related publications, and many had more than ten. Before annotation, each annotator received a one-hour training session.
Annotation proceeded in three stages:
Question annotation: annotators first saw only the abstract, introduction, and schematic diagram. They were assigned one of the five information-seeking scenarios, wrote a question, and drew one or two colored bounding boxes around relevant visual elements. This stage was designed to mimic early paper reading, where readers often inspect the overview figure and opening sections before reading the full paper.
Answer annotation: after question writing, annotators gained access to the full paper and reassessed whether the question was answerable. If answerable, they identified supporting sections or subsections; otherwise, the question was marked unanswerable. A second annotator then reviewed question quality, revised or discarded unclear or insufficiently challenging items, validated evidence, and wrote a free-form answer constrained to paper content.
Validation: a third annotator checked whether the question was meaningful and grammatical, whether diagram interpretation was genuinely required, whether bounding boxes were aligned with the question, whether the evidence was complete and accurate, and whether the answer was correct and free of outside knowledge.
The benchmark therefore does not rely on crowdsourcing. It is built as an expert-annotated scientific-document resource in which diagram grounding, answerability, evidence selection, and free-form answer writing are all controlled by researchers.
4. Dataset composition and split structure
The resulting benchmark contains 1,500 expert-annotated QA examples over 465 papers. Of these, 1,102 are answerable and 398 are intentionally unanswerable, meaning that 26.5% of the benchmark tests abstention and calibration directly.
| Statistic | Value |
|---|---|
| Scientific papers | 465 |
| QA examples | 1,500 |
| Answerable questions | 1,102 |
| Unanswerable questions | 398 |
| Schematic diagram caption length | median 57 / average 67 |
| Paper length | median 4,368 / average 4,565 |
| Question length | median 14.1 / average 13.8 |
| Answer length | median 59.0 / average 60.3 |
The answer-length statistics exclude unanswerable questions. The paper randomly divides the data into testmini with 500 examples for development validation and test with 1,000 examples for standard evaluation.
For the test set, the paper reports the following topic counts: Design Rationale 207, Implementation Details 212, Literature Background 201, Experimental Results 193, Other 187, and Unanswerable 225. These counts sum to 1,225 because unanswerable is treated as an orthogonal breakdown rather than a sixth mutually exclusive topic.
This composition has two methodological consequences. First, MISS-QA is not dominated by a single reasoning type; the five subsets are close in size. Second, abstention is sufficiently common that a model can neither ignore unanswerability nor treat it as a corner case.
5. Evaluation protocol and empirical results
MISS-QA evaluates 18 multimodal foundation models, grouped into proprietary and open-source systems. The proprietary set comprises OpenAI o4-mini, GPT-4.1, GPT-4.1-mini, GPT-4o, and Gemini-2.5-Flash. The open-source set comprises Qwen2.5-VL-72B, Qwen2-VL-72B, Qwen2.5-VL-7B, Qwen2-VL-7B, InternVL3-38B, InternVL3-8B, InternVL2.5-38B, InternVL2.5-8B, InternVL2-8B, Pixtral-12B, Mistral-Small-3.1-24B, Phi-3.5-Vision, and Phi-4-Multimodal (Zhao et al., 14 Jul 2025).
All models are prompted with a common instruction that frames the model as a computer science researcher, supplies the image, the schematic diagram caption, the paper context, and the question, and explicitly instructs the model to conclude with “I do not know.” when the answer is not supported by the paper. The evaluation metric is called accuracy, but it is not exact-match accuracy. Instead, MISS-QA uses an LLM-as-Judge protocol with GPT-4.1 as evaluator, assigning scores of 0, 0.5, or 1 by comparing the model response to the gold answer. Partial credit is therefore built into the benchmark.
Human expert performance was measured by asking two PhD candidates, one in NLP and one in CV, to solve 50 examples sampled as 10 from each subset of testmini within 5 hours. Their responses were graded with the same judging system, yielding 89.0% average accuracy.
The main leaderboard establishes a large human-model gap. The best overall model is o4-mini at 78.3 on the test set, followed closely by GPT-4.1 at 77.8. The strongest open-source model is Qwen2.5-VL-72B at 61.6. Human experts remain far ahead at 89.0%, which is a gap of 27.4 points relative to the best open-source model.
| System | Test accuracy |
|---|---|
| Human experts | 89.0 |
| o4-mini | 78.3 |
| GPT-4.1 | 77.8 |
| GPT-4.1-mini | 74.1 |
| Gemini-2.5-Flash | 67.3 |
| GPT-4o | 63.0 |
| Qwen2.5-VL-72B | 61.6 |
| InternVL3-38B | 60.4 |
| Mistral-Small-3.1-24B | 57.3 |
| Qwen2-VL-72B | 54.2 |
| Phi-3.5-Vision | 30.3 |
The paper also emphasizes within-family improvement: Qwen2.5-VL-72B exceeds Qwen2-VL-72B by 7.4 points, and InternVL3-38B exceeds InternVL2.5-38B by 8.9 points.
Category-level results reveal a non-uniform difficulty profile. Human experts scored 90.0 on Design Rationale, 85.0 on Implementation Details, 95.0 on Literature Background, 95.0 on Experimental Results, 80.0 on Other, and 85.0 on Unanswerable. Among proprietary models, o4-mini was strongest on most answerable categories, but its Unanswerable score was only 33.8. GPT-4.1 was notably stronger on Other (67.1) and Unanswerable (60.0). Gemini-2.5-Flash exhibited an unusual pattern: it was weaker on answerable science-content categories than the top proprietary models, but scored 87.2 on Other and 88.0 on Unanswerable.
Among open-source models, Qwen2.5-VL-72B and InternVL3-38B were close overall, but both were markedly weaker on unanswerable questions than on answerable categories. This asymmetry is one of the paper’s central findings.
6. Error structure, limitations, and research implications
Because 26.5% of MISS-QA is unanswerable, the benchmark directly exposes calibration failures. The paper’s qualitative conclusion is explicit: most models, except Gemini-2.5-Flash and some lower-performing Phi-family models, struggle with unanswerable questions and tend to answer anyway rather than abstain. The authors describe this as overconfidence. Human experts are much better at recognizing when the paper does not supply the requested information (Zhao et al., 14 Jul 2025).
The manual error analysis examines 100 randomly sampled error cases from the top-performing open-source models on testmini: 50 from Qwen2-VL-72B and 50 from InternVL2.5-38B. It identifies five main failure types.
Failure to interpret and contextualize schematic diagrams occurs when a model does not correctly decode what a highlighted box refers to or misses the structural relation among diagram elements.
Inability to retrieve relevant context occurs when a model broadly understands the topic but fails to locate the relevant section or sentence in the paper, producing generic or incomplete answers.
Reasoning error occurs when relevant visual and textual evidence is identified but the final explanation remains shallow or paper-nonspecific.
Overconfident response to unanswerable questions is a direct hallucination failure in which a model invents an explanation although the gold answer is explicitly that the question is unanswerable.
Overreliance on visual elements occurs when the model answers from isolated figure interpretation or parametric prior knowledge instead of consulting the paper text for the paper-specific explanation.
The paper also notes rarer failures involving the 1024-token output limit and refusals due to safety alignment.
These findings are summarized as weakness along three linked dimensions: diagram grounding, paper-context retrieval, and cross-modal reasoning and calibration. This suggests that progress on chart interpretation or table reading does not automatically transfer to schematic-overview understanding.
The benchmark’s stated limitations are also precise. First, it is mostly limited to AI-related arXiv papers, because the annotator pool consisted of AI experts. The paper notes that extension to other domains such as biology or medicine would improve generalizability. Second, although source papers were chosen to reduce memorization risk, contamination cannot be fully determined because most evaluated models do not disclose detailed pretraining data.
The broader significance of MISS-QA lies in the kind of scientific literacy it measures. It tests whether a model can use the overview figure as researchers often do: as a navigation device for the full paper. The reported results indicate that even strong multimodal foundation models remain substantially below expert humans on that capability. A plausible implication is that future systems will need tighter integration of diagram interpretation, long-document evidence retrieval, and calibrated abstention, rather than incremental improvements in OCR or generic visual-language alignment alone.