SciDiagram-Robust: Diagram QA Benchmark

• SciDiagram-Robust is a benchmark dataset designed for robust scientific diagram question-answering across both clean and perturbed images.
• It augments 3,500 QA pairs with programmatically generated degradations, including noise, blur, occlusion, and rotation to mimic real-world challenges.
• The dataset introduces evaluation metrics such as Clean Accuracy, Perturbation Robustness Score (PRS), and Visual Degradation Consistency (VDC) to measure model stability.

Searching arXiv for the cited papers and closely related robustness work on scientific diagram reasoning and multimodal robustness. SciDiagram-Robust is a benchmark dataset for evaluating the robustness of Large Vision-LLMs (LVLMs) on scientific diagram question answering under visually perturbed conditions. It is introduced as the first large-scale scientific diagram question-answering dataset specifically augmented with diverse, programmatically generated visual perturbations, and it is designed to expose how performance on clean diagrams can diverge from performance under noise, blur, occlusion, and related degradations that are common in real-world scientific documents (Zhou et al., 23 Aug 2025).

1. Definition and scope

SciDiagram-Robust extends scientific QA settings from clean-diagram evaluation to robustness evaluation. The dataset is specifically designed for assessing whether an LVLM can preserve reasoning quality when a scientific diagram is visually degraded, rather than merely achieving strong clean-image accuracy. In the formulation associated with the dataset, the central problem is robust diagram reasoning: answering the same scientific question correctly across clean and perturbed versions of a diagram (Zhou et al., 23 Aug 2025).

The dataset is described as the first large-scale scientific diagram question-answering dataset augmented with multiple programmatically generated visual degradations. Its intended role is not only comparative benchmarking but also the systematic analysis of failure modes under perturbation type and perturbation intensity. This suggests a shift from standard visual QA evaluation toward robustness-sensitive evaluation in which stability across multiple degraded views becomes a first-class criterion (Zhou et al., 23 Aug 2025).

SciDiagram-Robust is presented in conjunction with the Robust Diagram Reasoning (RDR) framework, but the dataset has an independent significance. It provides the clean and perturbed diagram variants, the associated questions and answers, and the evaluation setting required to study zero-shot robustness without fine-tuning on degraded diagrams (Zhou et al., 23 Aug 2025).

2. Source data, domains, and dataset composition

SciDiagram-Robust extends and enhances existing public scientific QA datasets, most notably ScienceQA and ChartQA. These datasets provide the base set of scientific diagrams and associated questions and answers. The resulting benchmark covers four scientific domains: Physics, Chemistry, Biology, and Geography, with 20 sub-topics within these domains (Zhou et al., 23 Aug 2025).

The diagram coverage is heterogeneous. The dataset includes standard scientific diagrams, flowcharts, graphs, plots, tables, and charts. This breadth is important because perturbation robustness on one visual format does not necessarily imply robustness on another. A plausible implication is that the benchmark is intended to prevent overly narrow conclusions drawn from a single diagram modality (Zhou et al., 23 Aug 2025).

The reported scale is 3,500 QA problems. For each question, there is one clean diagram and 10 perturbed versions, yielding 11 images per QA pair. The answer types are multiple-choice, fill-in-the-blank, and short-answer (Zhou et al., 23 Aug 2025).

Property	Value
QA problems	3,500
Domains	Physics, Chemistry, Biology, Geography
Per-question images	1 clean + 10 perturbed
Answer types	Multiple-choice, fill-in-the-blank, short-answer

The per-question image multiplicity is central to the benchmark design. Rather than treating each degraded image as an isolated test case, SciDiagram-Robust groups clean and perturbed versions of the same underlying diagram-question pair. This supports direct measurement of answer preservation and cross-perturbation consistency (Zhou et al., 23 Aug 2025).

3. Perturbation model and generation protocol

For each original diagram, up to 5 distinct types of visual perturbations are programmatically generated using image processing libraries such as OpenCV and PIL together with custom scripts. Each perturbation is produced at 3 different intensity levels. The perturbation types listed for the benchmark are Gaussian noise, salt-and-pepper noise, motion blur, local occlusion, and slight rotation (Zhou et al., 23 Aug 2025).

The dataset description also states that each clean diagram is paired with 10 unique perturbed versions. The perturbation design is intended to reflect common real-world degradation scenarios, including noise from scanning, blur from low resolution, and occlusion from overlapping content. Because the augmentations are systematic and intensity-controlled, SciDiagram-Robust enables fine-grained robustness analysis rather than only aggregate failure counting (Zhou et al., 23 Aug 2025).

The perturbations have distinct operational interpretations. Gaussian noise and salt-and-pepper noise target stochastic corruption; motion blur targets loss of sharp spatial detail; local occlusion blocks out parts of the image; slight rotation introduces geometric transformation. The dataset therefore tests both appearance corruption and structural misalignment. This suggests that robustness on SciDiagram-Robust depends on more than OCR resilience or simple denoising capacity (Zhou et al., 23 Aug 2025).

A hypothetical example in the source description uses a bar chart of average precipitation in four cities, then applies Gaussian noise, salt-and-pepper noise, vertical motion blur, left-side occlusion, and a 5-degree rotation while preserving the same question and evaluating answer consistency across variants (Zhou et al., 23 Aug 2025). The example is illustrative rather than a dataset statistic, but it clarifies the intended evaluation semantics.

4. Evaluation protocol and robustness metrics

SciDiagram-Robust uses a zero-shot evaluation paradigm: models are not fine-tuned on perturbed diagrams, and each model must process both clean and degraded diagrams for the same questions. This is explicitly framed as a realistic application scenario (Zhou et al., 23 Aug 2025).

The benchmark introduces two robustness metrics in addition to Clean Accuracy (CA). Clean Accuracy is the percentage of questions answered correctly on the original clean diagrams. Perturbation Robustness Score (PRS) measures the fraction of questions for which the model gives the correct answer across all perturbed versions, or after self-correction in the RDR setup. Visual Degradation Consistency (VDC) measures the proportion of perturbed answers that match the clean diagram answer, averaged over all questions (Zhou et al., 23 Aug 2025).

The formal definitions are:

$$\text{PRS} = \frac{1}{M} \sum_{j=1}^{M} \mathbb{I} \left( \forall i \in \{0, \dots, N_j\}, \mathcal{M}(D_{j,i}, Q_j) = GT_j \right) \times 100\%$$

where $M$ is the number of questions, $N_j$ is the number of perturbed versions for question $j$, $GT_j$ is the ground truth, and $\mathbb{I}$ is the indicator function (Zhou et al., 23 Aug 2025).

For VDC, the per-question consistency is defined as

$$\text{Consistency}_j = \frac{1}{N_j+1} \sum_{i=0}^{N_j} \mathbb{I}(A_{j,i} = A_{j,0})$$

and the dataset-level score is

$$\text{VDC} = \frac{1}{M} \sum_{j=1}^{M} \text{Consistency}_j \times 100\%$$

where $A_{j,0}$ is the answer from the clean diagram for question $j$ (Zhou et al., 23 Aug 2025).

These metrics distinguish three notions that are often conflated. CA measures standard task performance on clean inputs. PRS requires correctness across all perturbed views and is therefore a stringent robustness criterion. VDC measures answer stability relative to the clean prediction, even when the clean prediction itself may not be correct. This suggests that SciDiagram-Robust separates robustness-as-correctness from robustness-as-consistency (Zhou et al., 23 Aug 2025).

5. Relationship to the RDR framework

SciDiagram-Robust directly supports the Robust Diagram Reasoning framework. In the description of that framework, the dataset enables perturbation-sensitive encoding through multiple perturbed images of the same diagram, multi-view parallel reasoning over those variants, and consistency verification with self-correction when outputs disagree (Zhou et al., 23 Aug 2025).

At the core of RDR is an Adaptive Multi-View & Consistency Verification (AMCV) mechanism. The mechanism involves generating multiple perturbed versions of a diagram, performing parallel inference, and then applying a consistency-based self-correction loop. SciDiagram-Robust is structurally aligned with this approach because each QA item is paired with a clean diagram and multiple perturbed counterparts (Zhou et al., 23 Aug 2025).

The benchmark therefore serves two roles. First, it is an evaluation resource for testing robustness under controlled visual perturbations. Second, it is a substrate for inference-time strategies that exploit multiple degraded views and cross-view agreement. A plausible implication is that the dataset is particularly well suited to methods that treat perturbations as a source of epistemic triangulation rather than merely as adversarial nuisance (Zhou et al., 23 Aug 2025).

This orientation differs from approaches that focus on robustness through training-time adaptation alone. A related paper, Robust-R1, addresses robustness in Multimodal LLMs by explicitly modeling visual degradations through structured reasoning chains, combining supervised fine-tuning, reward-driven alignment, and dynamic reasoning depth scaling (Tang et al., 19 Dec 2025). The two works are not the same framework, but both treat degradation as an object of explicit modeling rather than as a hidden failure mode of the visual encoder.

6. Empirical findings and significance

SciDiagram-Robust is used to show that strong clean-diagram performance does not guarantee perturbation robustness. The reported results indicate that even state-of-the-art closed-source LVLMs such as GPT-4V exhibit substantial degradation under perturbed inputs, with Clean Accuracy 85.2% versus PRS 72.1% (Zhou et al., 23 Aug 2025).

The benchmark results reported for several models are as follows:

Model	CA	PRS / VDC
GPT-4V	85.2%	72.1% / 78.5%
Gemini Pro Vision	82.5%	69.8% / 75.3%
LLaVA-1.5-13B	78.9%	65.4% / 70.1%
RDR-LLaVA-13B	79.5%	74.5% / 81.2%

These results support several claims stated in the source material. All models lose significant accuracy under perturbations. RDR-LLaVA-13B outperforms GPT-4V and Gemini Pro Vision in robustness metrics, specifically PRS and VDC. The perturbation-type analysis further states that RDR achieves the greatest robustness gains for more difficult perturbations, including local occlusion and salt-and-pepper noise, and that the improvement increases as perturbations become more severe (Zhou et al., 23 Aug 2025).

The significance of these findings lies in what the benchmark reveals about evaluation practice. Prior benchmarks such as ScienceQA and ChartQA use clean diagrams, whereas SciDiagram-Robust systematically augments diagrams with diverse and realistic visual perturbations and enables fine-grained analysis of model failures under each perturbation type and intensity (Zhou et al., 23 Aug 2025). This suggests that clean-only benchmarks can overestimate deployable competence in scientific-document settings.

7. Position within robust multimodal reasoning research

SciDiagram-Robust occupies a specific niche within robustness research: scientific diagram reasoning under visual perturbation. Its focus is narrower than general multimodal robustness benchmarks, but this narrowness is methodological rather than limiting. Scientific diagrams combine symbolic, textual, spatial, and relational content, so perturbation-induced failures can arise from disrupted structure as well as degraded appearance (Zhou et al., 23 Aug 2025).

Within this landscape, SciDiagram-Robust is best understood as a benchmark that makes robustness measurable in a domain where diagram interpretation is often structurally demanding. The dataset does not merely add corrupted images; it defines an evaluation problem in which maintaining correct reasoning across controlled perturbations is itself the target of measurement (Zhou et al., 23 Aug 2025).

A useful clarification is that SciDiagram-Robust is distinct from general image-corruption benchmarks and also distinct from work on robustness in other uses of the word “diagram.” Its subject is scientific diagram QA for LVLMs. Its characteristic contributions are domain-diverse diagram coverage, systematic perturbation generation, grouped clean/perturbed evaluation, and the introduction of PRS and VDC as robustness-sensitive metrics (Zhou et al., 23 Aug 2025).

In that sense, SciDiagram-Robust functions as both a diagnostic instrument and a methodological constraint. It diagnoses brittleness that clean accuracy can conceal, and it constrains future methods to address robustness in a setting where visual degradation is explicit, measurable, and operationally relevant.