---
title: 'MISS-QA: Multimodal Diagram Benchmark'
url: https://www.emergentmind.com/topics/miss-qa
type: topic
---

# MISS-QA: Multimodal Diagram Benchmark

Searching arXiv for MISS-QA and a few related scientific-paper QA benchmarks to ground the article with current paper identifiers.
{"query":"arXiv 2507.10787 MISS-QA schematic diagrams scientific papers", "max_results": 5}
{"query":"QASPER question answering scientific papers arXiv", "max_results": 10}
{"query":"QASA scientific papers question answering arXiv", "max_results": 10}
MISS-QA is a benchmark for **Multimodal Information-Seeking over Scientific papers** that evaluates whether multimodal foundation models can interpret **schematic diagrams** in research papers and use them together with paper text to answer realistic, free-form, information-seeking questions. It was introduced as the first benchmark specifically targeting this capability, with **1,500 expert-annotated examples** drawn from **465 scientific papers** and an evaluation suite spanning **18 frontier multimodal foundation models** plus human experts. The benchmark is motivated by a gap between **text-only** scientific-paper QA and multimodal scientific QA settings that focus mainly on **charts, tables, or experimental-result figures**, rather than on overview diagrams that summarize methods, workflows, modules, and conceptual relations [2507.10787].

## 1. Conceptual scope and distinctiveness

MISS-QA is defined around a specific document modality: the **schematic overview diagram** that typically appears as the **first or second figure** of a scientific paper. In the benchmark, these figures are chosen because they usually present a research overview, system pipeline, architecture, or methodological framework. The task is not figure captioning, OCR, or generic visual question answering in isolation. Instead, a model must use the diagram to identify the referenced component and then consult the broader paper context to answer a question whose resolution often depends on motivations, implementation details, literature grounding, experimental implications, or limitations [2507.10787].

The paper positions MISS-QA against two established lines of work. On one side are **scientific-paper QA benchmarks such as QASPER and QASA**, which are described as essentially **text-only**. On the other side are multimodal scientific QA or figure benchmarks such as **ArXivQA, MMSci, CharXiv, and M3SciQA**, which are described as focusing mostly on **charts/tables or experimental-result figures**. MISS-QA isolates a different capability: understanding diagrams that summarize a paper’s method and then aligning those diagram elements with the full document.

A central design choice is the use of **colored bounding boxes**. Each question refers to one or two highlighted visual elements using placeholders such as “the module highlighted by the red bounding box.” This prevents a shortcut in which a model answers from the wording of the question alone. Rather than naming a component directly, the question forces diagram inspection first and only then paper-context retrieval.

Two misconceptions are explicitly addressed by the benchmark design. First, MISS-QA is not a pure vision benchmark: the answer is often not visually explicit in the figure alone. Second, it is not reducible to long-context text QA: the question formulation is intended to require identification of the relevant diagram component before the text can be used.

## 2. Task formulation and question taxonomy

Formally, the task takes a schematic diagram `diag`, a question `q`, and the main sections of the paper `p`, and requires a model to produce a free-form answer `a`. In plain terms, the model conditions jointly on the **visual overview figure** and the **paper text** and generates the most probable answer. The paper describes this as a generative information-seeking task rather than a classification problem.

The benchmark organizes questions into five information-seeking subsets. These subsets are intended to reflect how researchers use overview diagrams as entry points into papers.

| Subset | Focus |
|---|---|
| Design rationale | Motivation or conceptual reason for a highlighted design choice |
| Implementation details | Technical interaction between highlighted modules or procedures |
| Literature background | Relation to prior work or methodological traditions |
| Experimental results | What results imply about a highlighted module’s role or effect |
| Others | Limitations, alternatives, ethics, or related aspects not covered above |

These categories are not merely topical labels. They encode different reasoning demands. **Design rationale** requires recovering why a module exists; **implementation details** requires technical interaction analysis; **literature background** requires placing a module in methodological lineage; **experimental results** requires connecting the highlighted element to empirical findings; and **Others** captures residual but still paper-grounded questions.

The benchmark also includes **unanswerable questions** as an explicit design axis. This is not treated as a separate information-seeking category in the original taxonomy; rather, answerability is orthogonal to the five subsets. A model is therefore expected not only to answer grounded questions but also to abstain and say **“I do not know”** when the paper does not contain the needed information. This makes calibration and abstention part of the benchmark’s intended scope.

## 3. Corpus construction and expert annotation

The data construction pipeline is structured around three stated desiderata: the benchmark should reflect **real-world research scenarios**, require actual **schematic diagram interpretation**, and cover **diverse reasoning types**. Source papers were collected from arXiv in AI-related categories: **Artificial Intelligence**, **Computation and Language**, **Machine Learning**, **Computer Vision and Pattern Recognition**, and **Information Retrieval**. The paper selection window was **July 1 to November 30, 2024**, chosen to be later than the cutoff dates for most open-source model pretraining corpora in order to reduce contamination concerns [2507.10787].

Filtering was manual and expert-driven. **Five expert annotators**, one per subfield, excluded non-research papers such as surveys or dissertations, papers whose first two figures were unsuitable schematic diagrams, and papers judged low quality or affected by HTML parsing errors. After filtering, the benchmark retained **465 scientific papers**. For each paper, annotators labeled the **first figure identified as a schematic diagram**, and that figure became the basis for QA annotation.

The annotation workforce consisted of **16 experts** in AI-related fields, including research scientists, postdocs, and PhD students across ML, CV, NLP, IR/NLP, CV/NLP, and DM. Every annotator had authored at least **three AI-related publications**, and many had more than ten. Before annotation, each annotator received a **one-hour training session**.

Annotation proceeded in three stages:

**Question annotation**: annotators first saw only the **abstract**, **introduction**, and **schematic diagram**. They were assigned one of the five information-seeking scenarios, wrote a question, and drew one or two colored bounding boxes around relevant visual elements. This stage was designed to mimic early paper reading, where readers often inspect the overview figure and opening sections before reading the full paper.

**Answer annotation**: after question writing, annotators gained access to the **full paper** and reassessed whether the question was answerable. If answerable, they identified supporting sections or subsections; otherwise, the question was marked **unanswerable**. A **second annotator** then reviewed question quality, revised or discarded unclear or insufficiently challenging items, validated evidence, and wrote a **free-form answer** constrained to paper content.

**Validation**: a **third annotator** checked whether the question was meaningful and grammatical, whether diagram interpretation was genuinely required, whether bounding boxes were aligned with the question, whether the evidence was complete and accurate, and whether the answer was correct and free of outside knowledge.

The benchmark therefore does not rely on crowdsourcing. It is built as an expert-annotated scientific-document resource in which diagram grounding, answerability, evidence selection, and free-form answer writing are all controlled by researchers.

## 4. Dataset composition and split structure

The resulting benchmark contains **1,500 expert-annotated QA examples** over **465 papers**. Of these, **1,102** are answerable and **398** are intentionally unanswerable, meaning that **26.5%** of the benchmark tests abstention and calibration directly.

| Statistic | Value |
|---|---|
| Scientific papers | 465 |
| QA examples | 1,500 |
| Answerable questions | 1,102 |
| Unanswerable questions | 398 |
| Schematic diagram caption length | median 57 / average 67 |
| Paper length | median 4,368 / average 4,565 |
| Question length | median 14.1 / average 13.8 |
| Answer length | median 59.0 / average 60.3 |

The answer-length statistics exclude unanswerable questions. The paper randomly divides the data into **testmini** with **500 examples** for development validation and **test** with **1,000 examples** for standard evaluation.

For the test set, the paper reports the following topic counts: **Design Rationale 207**, **Implementation Details 212**, **Literature Background 201**, **Experimental Results 193**, **Other 187**, and **Unanswerable 225**. These counts sum to **1,225** because **unanswerable** is treated as an orthogonal breakdown rather than a sixth mutually exclusive topic.

This composition has two methodological consequences. First, MISS-QA is not dominated by a single reasoning type; the five subsets are close in size. Second, abstention is sufficiently common that a model can neither ignore unanswerability nor treat it as a corner case.

## 5. Evaluation protocol and empirical results

MISS-QA evaluates **18 multimodal foundation models**, grouped into proprietary and open-source systems. The proprietary set comprises **OpenAI o4-mini**, **GPT-4.1**, **GPT-4.1-mini**, **GPT-4o**, and **Gemini-2.5-Flash**. The open-source set comprises **Qwen2.5-VL-72B**, **Qwen2-VL-72B**, **Qwen2.5-VL-7B**, **Qwen2-VL-7B**, **InternVL3-38B**, **InternVL3-8B**, **InternVL2.5-38B**, **InternVL2.5-8B**, **InternVL2-8B**, **Pixtral-12B**, **Mistral-Small-3.1-24B**, **Phi-3.5-Vision**, and **Phi-4-Multimodal** [2507.10787].

All models are prompted with a common instruction that frames the model as a computer science researcher, supplies the **image**, the **schematic diagram caption**, the **paper context**, and the **question**, and explicitly instructs the model to conclude with **“I do not know.”** when the answer is not supported by the paper. The evaluation metric is called **accuracy**, but it is not exact-match accuracy. Instead, MISS-QA uses an **LLM-as-Judge** protocol with **GPT-4.1** as evaluator, assigning scores of **0**, **0.5**, or **1** by comparing the model response to the gold answer. Partial credit is therefore built into the benchmark.

Human expert performance was measured by asking **two PhD candidates**, one in NLP and one in CV, to solve **50 examples** sampled as **10 from each subset of testmini** within **5 hours**. Their responses were graded with the same judging system, yielding **89.0%** average accuracy.

The main leaderboard establishes a large human-model gap. The best overall model is **o4-mini** at **78.3** on the test set, followed closely by **GPT-4.1** at **77.8**. The strongest open-source model is **Qwen2.5-VL-72B** at **61.6**. Human experts remain far ahead at **89.0%**, which is a gap of **27.4 points** relative to the best open-source model.

| System | Test accuracy |
|---|---:|
| Human experts | 89.0 |
| o4-mini | 78.3 |
| GPT-4.1 | 77.8 |
| GPT-4.1-mini | 74.1 |
| Gemini-2.5-Flash | 67.3 |
| GPT-4o | 63.0 |
| Qwen2.5-VL-72B | 61.6 |
| InternVL3-38B | 60.4 |
| Mistral-Small-3.1-24B | 57.3 |
| Qwen2-VL-72B | 54.2 |
| Phi-3.5-Vision | 30.3 |

The paper also emphasizes within-family improvement: **Qwen2.5-VL-72B** exceeds **Qwen2-VL-72B** by **7.4 points**, and **InternVL3-38B** exceeds **InternVL2.5-38B** by **8.9 points**.

Category-level results reveal a non-uniform difficulty profile. Human experts scored **90.0** on Design Rationale, **85.0** on Implementation Details, **95.0** on Literature Background, **95.0** on Experimental Results, **80.0** on Other, and **85.0** on Unanswerable. Among proprietary models, **o4-mini** was strongest on most answerable categories, but its **Unanswerable** score was only **33.8**. **GPT-4.1** was notably stronger on **Other** (**67.1**) and **Unanswerable** (**60.0**). **Gemini-2.5-Flash** exhibited an unusual pattern: it was weaker on answerable science-content categories than the top proprietary models, but scored **87.2** on **Other** and **88.0** on **Unanswerable**.

Among open-source models, **Qwen2.5-VL-72B** and **InternVL3-38B** were close overall, but both were markedly weaker on unanswerable questions than on answerable categories. This asymmetry is one of the paper’s central findings.

## 6. Error structure, limitations, and research implications

Because **26.5%** of MISS-QA is unanswerable, the benchmark directly exposes calibration failures. The paper’s qualitative conclusion is explicit: **most models, except Gemini-2.5-Flash and some lower-performing Phi-family models, struggle with unanswerable questions** and tend to answer anyway rather than abstain. The authors describe this as **overconfidence**. Human experts are much better at recognizing when the paper does not supply the requested information [2507.10787].

The manual error analysis examines **100 randomly sampled error cases** from the top-performing open-source models on **testmini**: **50 from Qwen2-VL-72B** and **50 from InternVL2.5-38B**. It identifies five main failure types.

**Failure to interpret and contextualize schematic diagrams** occurs when a model does not correctly decode what a highlighted box refers to or misses the structural relation among diagram elements.

**Inability to retrieve relevant context** occurs when a model broadly understands the topic but fails to locate the relevant section or sentence in the paper, producing generic or incomplete answers.

**Reasoning error** occurs when relevant visual and textual evidence is identified but the final explanation remains shallow or paper-nonspecific.

**Overconfident response to unanswerable questions** is a direct hallucination failure in which a model invents an explanation although the gold answer is explicitly that the question is unanswerable.

**Overreliance on visual elements** occurs when the model answers from isolated figure interpretation or parametric prior knowledge instead of consulting the paper text for the paper-specific explanation.

The paper also notes rarer failures involving the **1024-token output limit** and refusals due to **safety alignment**.

These findings are summarized as weakness along three linked dimensions: **diagram grounding**, **paper-context retrieval**, and **cross-modal reasoning and calibration**. This suggests that progress on chart interpretation or table reading does not automatically transfer to schematic-overview understanding.

The benchmark’s stated limitations are also precise. First, it is mostly limited to **AI-related arXiv papers**, because the annotator pool consisted of AI experts. The paper notes that extension to other domains such as **biology** or **medicine** would improve generalizability. Second, although source papers were chosen to reduce memorization risk, contamination cannot be fully determined because most evaluated models do not disclose detailed **pretraining data**.

The broader significance of MISS-QA lies in the kind of scientific literacy it measures. It tests whether a model can use the overview figure as researchers often do: as a navigation device for the full paper. The reported results indicate that even strong multimodal foundation models remain substantially below expert humans on that capability. A plausible implication is that future systems will need tighter integration of **diagram interpretation**, **long-document evidence retrieval**, and **calibrated abstention**, rather than incremental improvements in OCR or generic visual-language alignment alone.

Source: https://www.emergentmind.com/topics/miss-qa