DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

Abstract: Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics, and scientific diagrams. Recent vision-LLMs (VLMs) often achieve high answer accuracy on these tasks, yet correct answers do not guarantee that models ground their reasoning in the diagram regions that support the prediction. Models may instead rely on textual correlations or dataset artifacts without identifying the visual evidence required to verify the answer. This limitation prevents reliable evaluation of diagram reasoning and reduces interpretability. We introduce DRAGON, a benchmark for evaluating evidence-grounded visual reasoning in diagrams. Given a diagram, a question, and the correct answer, a model must predict bounding boxes that correspond to the visual elements required to justify the answer. These evidence regions may include answer-bearing components, textual labels, legends, axes, connectors, and other supporting structures involved in the reasoning process. The DRAGON dataset contains 11,664 annotated question instances collected from six diagram QA datasets: ChartQA, Circuit-VQA, InfographicsVQA, MapIQ, MapWise, and AI2D. We release a 2,445-instance benchmark test set with human-verified reasoning evidence annotations and a standardized evaluation framework. We evaluate eight recent VLMs and analyze their ability to localize reasoning evidence across diverse diagram domains. DRAGON enables systematic evaluation of diagram reasoning and supports future research on models that ground their predictions in visual evidence.

• The paper presents DRAGON, a benchmark that requires models to localize all supporting evidence in diagrams to justify answers.
• It introduces three prompting strategies—EDGE, SAGE, and VERGE—to assess candidate selection, bounding box localization, and evidence refinement.
• Empirical results reveal that high answer accuracy does not ensure complete evidence grounding, highlighting the need for improved model reasoning fidelity.

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

Motivation and Problem Setting

Diagram question answering (DQA) tasks require models to integrate information across structured visual representations such as charts, maps, scientific illustrations, circuits, and infographics. Despite significant advances in VLMs, evaluation protocols for DQA remain inadequate: models frequently achieve high answer accuracy, yet there is no guarantee their reasoning is sound or that answers are visually grounded. Prior work demonstrates that VLMs can exploit textual correlations and data artifacts to answer questions, often without referencing the specific diagram regions that a human would require to verify an answer. This undermines interpretability, trustworthiness, and faithfulness, which are critical for scientific, educational, and technical applications.

The DRAGON benchmark directly addresses this evaluative deficiency. For a given diagram, question, and gold answer, the evidence-grounded reasoning task requires models to predict bounding boxes localizing all visual regions needed to justify the answer: not only answer-bearing regions, but also associated labels, legends, axes, connectors, or supporting diagram structures.

Figure 1: Overview of the DRAGON benchmark construction and evaluation pipeline with three stages: data generation and annotation, grounded reasoning via prompting, and extraction/evaluation of visual regions.

Benchmark Construction and Annotation Protocol

DRAGON aggregates and annotates 11,664 question instances from six diverse DQA datasets—AI2D, ChartQA, Circuit-VQA, InfographicsVQA, MapIQ, and MapWise. The construction protocol standardizes the annotation of visual evidence, regardless of the heterogeneity in underlying sources and diagram modalities.

For domains lacking predefined region-level annotations, DRAGON employs template-based region transfer and homography estimation to propose candidate evidence boxes, which are then verified, edited, or supplemented by expert annotators using the DIAGRAMS interface. Annotators are instructed to ground all minimal visual regions required to support inference—particularly in reasoning scenarios demanding intermediate comparisons, multi-hop steps, or spatial-localization (e.g., comparative analysis over map neighbors, multi-component identification in circuits).

Consensus is ensured via multi-reviewer verification and arbitration; reliability is quantified through inter-annotator agreement (with criteria for complete and core evidence alignment).

Prompting Strategies and Model Evaluation

To systematically examine model limitations, DRAGON defines three prompting regimes:

• EDGE (Evidence Detection via Grounding): Single-step direct grounding.
• SAGE (Select and Ground Evidence): Two-stage prompting with explicit identification of visual elements followed by coordinate prediction.
• VERGE (Verify and Refine Grounded Evidence): Adds a self-correction/verification step to encourage revision and augmentation of evidence sets.

This separation isolates the contributions of candidate selection, bounding box localization, and iterative refinement to total grounding performance.

DRAGON benchmarks eight leading VLMs—including Claude Opus 4.6, Gemini 3 Pro, Kimi K2.5, and major open-weight systems. Outputs are measured by Max Pairwise IoU (MP$_{\mathrm{IoU}}$), Grounding IoU (G$_{\mathrm{IoU}}$), and F1 over bounding boxes, ensuring precise separation between region localization and reasoning chain coverage.

Evidence-Grounded Reasoning: Qualitative Analysis

DRAGON exposes the structure and complexity required for authentic faithful diagram reasoning. Successful cases show stringent alignment with human-justified reasoning, where the model must ground both answer and supporting regions:

Figure 2: AI2D food-web diagram illustrating successful evidence-grounded reasoning: answer organisms are green, supporting food source in red.

Figure 3: ChartQA stacked bar illustrating year/segment-level localization of compared values and answer regions.

Figure 4: CircuitVQA example with correct grounding of all resistor components required for counting-based reasoning.

The evaluation pipeline provides granular failure analysis. Models frequently mislocalize evidence: selecting prominent but irrelevant regions, focusing on legends instead of instance regions, or omitting required multi-hop support (Figures 8–11). Conversely, DRAGON includes successful grounding cases (Figures 12–14) where models correctly isolate relevant objects or annotations.

Empirical Findings

Model and Prompt Sensitivity

Grounding performance remains weak across most domains:

• Best F1 reaches 21.8 (Claude Opus 4.6, MapIQ) and otherwise remains below 15 for most closed-source models.
• Open-weight models (e.g., Qwen3.5-35B-A3B, Gemma3-27B-IT, InternVL3.5-38B) exhibit a pronounced performance gap, rarely achieving F1 $>$ 7 (Table~\ref{tab:dataset_model}).

Closed-source models routinely outperform open-weight alternatives—prompting strategies (EDGE, SAGE, VERGE) provide moderate improvement in localization (MP$_\mathrm{IoU}$), but have only limited effect on complete evidence coverage (F1). Importantly, strong answer prediction does not entail sound evidence-grounded reasoning—models often localize a correct answer but fail to cover the full chain of visual justification.

Domain-Specific Difficulty

Performance degrades markedly in structure-dense, relationally complex domains (CircuitVQA, InfographicsVQA), where maximal F1 scores are typically below 6. The models are more effective in schematic or clearly annotated settings (MapIQ, AI2D) than in diagrams requiring multi-component count or cross-reference.

Localization vs. Evidence Coverage

Consistently, MP$_\mathrm{IoU}$ substantially exceeds G$_\mathrm{IoU}$ and F1: models can often find an approximately relevant region, but almost never recover all the visual evidence required for minimal justifiability. Prompting can improve coarse localization but does not close this gap.

Failure Modes

Common failure cases include:

• Grounding general regions rather than answer-specific labels or components (Figure 5)
• Ignoring intermediate steps (e.g., not isolating objects for comparison in maps, Figure 6)
• Failing to resolve symbolic or relational components requiring cross-reference (Figures 10–11)

Practical and Theoretical Implications

Practically, DRAGON makes the "reasoning faithfulness gap" in DQA machine-tractable. It enables standardized, multi-domain, large-scale diagnosis of VLMs' grounding behaviors. The performance gap—especially between closed and open models—emphasizes the need for explicit evidence-centric benchmarks in deployment-critical settings (science, engineering, education), where justified reasoning is essential.

Theoretically, DRAGON demonstrates that high answer accuracy is not sufficient for generalizable, trustworthy multimodal reasoning; explicit training and architectural alignment towards evidence-grounded inference is required. Performance differences across prompting strategies and model families suggest architectural improvements—possibly architectural disentanglement between entity segmentation, answer extraction, and reasoning chain synthesis—may yield stronger improvements than prompting or scaling alone.

Figure 7: InfographicsVQA example of correct answer and supporting numerical-statistical evidence grounding.

Figure 8: MapWise sample illustrating spatial-reference alignment and supporting region grounding.

Figure 9: MapIQ cartogram, evidence selection for comparative attribute reasoning.

Limitations and Future Prospects

DRAGON adopts bounding boxes as its annotation primitive, which, while standardized and practical, cannot capture fine-grained, nonrectangular, or intricate diagram regions with perfect fidelity. Human curation introduces some subjectivity despite detailed guidelines and multi-stage arbitration. The benchmark currently focuses on evaluation rather than large-scale training, but this design choice ensures its role as a standardized testbed rather than a target for overfitting.

Future avenues include expanding annotation modalities (e.g., segmentation masks, hierarchical region graphs), enabling reinforcement learning with evidence-aligned rewards, and extending DRAGON to new domains with higher compositional complexity or novel symbol systems. Importantly, DRAGON can serve as a substrate for research in interpretable reasoning, faithful multimodal chain-of-thought, and robust model auditing.

Conclusion

DRAGON establishes a rigorous, multi-domain benchmark for evaluating evidence-grounded visual reasoning in DQA settings. Empirical results highlight that current VLMs—regardless of scale or prompt engineering—fall far short in producing comprehensive, human-verifiable reasoning evidence. As a result, DRAGON reframes progress in DQA and VLMs towards explicit alignment between visual grounding and answer justification, providing an indispensable resource for future research into interpretable, trustworthy multimodal AI.

Citation: "DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams" (2604.25231)