Robust Diagram Reasoning (RDR)

• Robust Diagram Reasoning (RDR) is a framework that interprets, parses, and reasons over diagrams while maintaining stability against noise, blur, and geometric distortions.
• It leverages structured representations such as Diagram Parse Graphs and dynamic graph networks to ground visual evidence and facilitate multi-modal integration.
• Evaluation benchmarks and adaptive multi-view consistency measures demonstrate its effectiveness in tackling perturbations and the inherent arbitrariness of diagram layouts.

Robust Diagram Reasoning (RDR) is the study of methods and evaluation protocols for interpreting, parsing, and reasoning over diagrams in ways that remain reliable under visual perturbations, heterogeneous layouts, and multimodal ambiguity. In recent work, the term is used explicitly for a framework that targets scientific diagram question answering under noise, blur, occlusion, and geometric distortions, while adjacent research addresses the same robustness objective through structured parsing, evidence grounding, formal languages, and graph-based intermediate representations (Zhou et al., 23 Aug 2025, Iyengar et al., 28 Apr 2026, Wang et al., 13 Apr 2026).

1. Problem setting and scope

Diagram reasoning differs from conventional image understanding because diagrams contain visual and textual information in an abstract and integrated way, and because their layouts are arbitrary rather than governed by photographic regularities. Kim et al. characterize this as a setting in which proper solutions for automatically understanding diagrams have been difficult to develop due to multi-modality and arbitrariness of layouts, while Kembhavi et al. frame the task as identifying the structure of a diagram and the semantics of its constituents and their relationships (Kim et al., 2017, Kembhavi et al., 2016).

Recent robustness-oriented work makes the failure modes more explicit. The RDR framework for scientific diagrams starts from the observation that practical deployment of LVLMs is hindered by a critical lack of robustness to common visual perturbations such as noise, blur, and occlusions, and that existing evaluation benchmarks largely overlook this challenge (Zhou et al., 23 Aug 2025). In geometry, the central bottleneck is described differently but compatibly: MLLMs continue to struggle with geometric reasoning primarily due to the perception bottleneck regarding fine-grained visual elements (Wang et al., 13 Apr 2026). In software architecture diagrams, the best-performing model on SADU achieves only $70.18\%$ accuracy, while gpt-4o-mini achieves $17.77\%$, indicating that diagram reasoning and visual relation grounding remain weak even for strong general-purpose VLMs (Ouyang et al., 5 Apr 2026).

A recurring theme across these lines of work is that answer correctness alone is insufficient. DRAGON is explicitly motivated by the fact that models may achieve high answer accuracy on diagram QA while relying on textual correlations or dataset artifacts without identifying the visual evidence required to verify the answer (Iyengar et al., 28 Apr 2026). This shifts RDR from a narrow concern with end-task accuracy to a broader concern with stability, faithfulness, interpretability, and grounding.

2. Structured representations for robust reasoning

A major strand of RDR research uses explicit intermediate structure to convert diagrams from pixel arrays into objects, relations, and formal descriptions. Kembhavi et al. introduced Diagram Parse Graphs (DPGs) as a unified graph-based formalism for the syntactic structure of diagrams, with nodes for blobs, text boxes, arrow tails, and arrow heads, and edges for 10 detailed relationship types; syntactic parsing is then DPG inference, and semantic interpretation is performed in diagram question answering through DPG-based attention (Kembhavi et al., 2016).

Kim et al. extended the graph-centric approach with the Dynamic Graph Generation Network (DGGN), embedded in a unified diagram-parsing framework. DGGN combines an object detector with GRU-based relational reasoning and a Dynamic Adjacency Tensor Memory (DATM), which stores hidden states and relational probabilities for candidate edges. The key idea is that message aggregation over this memory makes relation inference less sensitive to arbitrary candidate ordering and more suitable for diagrams whose topology is not known a priori (Kim et al., 2017).

In geometry, recent work replaces generic graph structure with a formal symbolic language. "Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language" introduces a unified formal language spanning plane and solid geometry, with syntax such as point A, line A B C, plane A B C D, AB \perp to CD on X, and solid Cube ABCD-A_{1}B_{1}C_{1}D_{1}. The language is hierarchical: 3D solids are explicitly described as hierarchical compositions of 2D primitives, allowing transfer from plane geometric perception to spatial tasks (Wang et al., 13 Apr 2026).

A more general formalization predates neural diagram parsing. Banerjee and Chandrasekaran model diagrammatic perceptions and actions as quantified constraint satisfaction problems over points, curves, and regions. In their framework, a spatial problem is written as

$$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$

and complex problems are decomposed into similar subproblems whose symbolic solutions can be reused by variable mapping (Banerjee et al., 2014). This line of work is not a perturbation benchmark, but it directly addresses robustness in the sense of domain-independent execution of perceptions and actions.

At a different level of abstraction, "On the Diagram of Thought" models reasoning itself as construction and navigation of a DAG within a single LLM, with learned role-specific tokens such as <proposer>, <critic>, and <summarizer>. The paper formalizes the reasoning DAG as a diagram in a topos and treats the synthesis step as a colimit (Zhang et al., 2024). This is not visual diagram reasoning, but it is relevant as an internal analogue of diagram-structured reasoning and faithfulness.

3. Robustness mechanisms and architectural patterns

The most explicit RDR mechanism is Adaptive Multi-View & Consistency Verification (AMCV). Given a diagram $D_0$ and question $Q$, AMCV generates perturbed views

$$D_i = p_{(i \bmod K)+1}(D_0, \text{intensity}_i), \quad i=1..N,$$

runs parallel inference

$A_i = \mathcal{M}(D_i, Q), \quad i=0..N,$

and computes a consistency score

$$C_Q = \frac{1}{N+1} \sum_{i=0}^{N} \mathbb{I}(A_i = A_{\text{mode}}).$$

If $C_Q$ falls below a threshold $\tau$, the model enters a self-correction step; otherwise it returns the modal answer. This mechanism operationalizes robustness as agreement across degraded views rather than confidence in a single pass (Zhou et al., 23 Aug 2025).

A second architectural pattern is modularization of reasoning into causally linked substeps. FaiRR decomposes deductive reasoning into rule selection, fact selection, and knowledge composition, with independently modeled transformer components. The paper argues that this modular process ensures model faithfulness by assured causal relation from the proof step to the inference reasoning, and the supplied summary states that, for robust diagram reasoning, such staged pipelines could similarly enhance faithfulness, error traceability, and resistance to distraction or out-of-distribution perturbations (Sanyal et al., 2022).

A third pattern is hierarchical specialization with global consistency checks. MACReD applies this to chemical reaction diagrams through a planning layer, a perception layer with specialized agents for molecules, arrows, and text, and a reasoning layer that fuses a spatial-semantic graph, a chemistry-aware graph, and a VLM-induced graph. The fused score is defined by weighted edge aggregation,

$17.77\%$0

followed by pruning and global reaction inference over connected components (Tang et al., 27 May 2026).

A plausible implication is that robustness in diagram reasoning increasingly depends on selective trust in intermediate computations rather than unconditional propagation of all latent signals. GateKD, although developed for reasoning distillation rather than diagrams, formalizes this principle with confidence-gated soft supervision, gated hidden-state evolution, and reliability-filtered attention distillation. Its sequence-level confidence gate is

$17.77\%$1

and the paper argues that gating suppresses hallucination transfer and stabilizes student reasoning (Sermsri et al., 13 May 2026).

4. Benchmarks and evaluation methodology

RDR has become benchmark-driven. Representative resources now probe perturbation robustness, formal parsing quality, evidence grounding, and domain-specific diagram comprehension across scientific, geometric, software, and educational settings (Zhou et al., 23 Aug 2025, Wang et al., 13 Apr 2026, Iyengar et al., 28 Apr 2026, Ouyang et al., 5 Apr 2026, Kembhavi et al., 2016).

Benchmark or dataset	Scope	Main evaluation signals
SciDiagram-Robust	3,500 scientific diagram QA samples with 10 perturbed versions per diagram	Clean Accuracy, PRS, VDC
GDP-29K	28,977 geometry diagrams: 19,965 plane and 8,917 solid	aggregate parsing accuracy, PPR, SA
DRAGON	11,664 annotated QA instances; 2,445-instance benchmark test set	$17.77\%$2, $17.77\%$3, box-level F1
SADU	154 software architecture diagrams and 2,431 QA tasks	EM, MAE, $17.77\%$4, Precision/Recall/F1
AI2 Diagrams	over 5,000 diagrams and 15,000+ questions	JIG, QA accuracy

SciDiagram-Robust uses five perturbation types—Gaussian noise, salt-and-pepper noise, motion blur, local occlusion, and slight rotation—each at three intensity levels, with 10 distinct perturbed versions per original diagram (Zhou et al., 23 Aug 2025). Its two central metrics are Perturbation Robustness Score (PRS), which measures the proportion of QA samples for which the model produces the correct answer across perturbed views or after self-correction, and Visual Degradation Consistency (VDC), which measures stability relative to the clean answer. The VDC definition is

$17.77\%$5

with

$17.77\%$6

The distinction matters because high VDC may reflect stable but incorrect behavior (Zhou et al., 23 Aug 2025).

DRAGON measures evidence-grounded reasoning rather than answer selection. Given a diagram, question, and correct answer, the model must predict bounding boxes for the visual regions needed to justify the answer. Its metrics include Max Pairwise IoU,

$17.77\%$7

and Grounding IoU,

$17.77\%$8

along with thresholded hit rates and box-level precision, recall, and F1 (Iyengar et al., 28 Apr 2026).

SADU evaluates VLMs on software architecture diagrams as structured software engineering artifacts rather than generic images. It combines rule-based exact match, LLM-as-a-judge evaluation, counting-specific metrics such as MAE and $17.77\%$9, retrieval-specific precision/recall/F1, subset and superset rates, and average missing or spurious elements (Ouyang et al., 5 Apr 2026). Earlier datasets established the basic syntax of evaluation: AI2 Diagrams introduced Jaccard Index for Graphs (JIG) for DPG inference, while the diagrammatic reasoning dataset of Saha et al. evaluates multiple-choice completion accuracy over rotation, counting, scaling, and other-type problems (Kembhavi et al., 2016, Ahmed et al., 2019).

5. Empirical findings across domains

The perturbation benchmark results that motivate RDR are numerically sharp. On SciDiagram-Robust, GPT-4V has Clean Accuracy $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$0, PRS $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$1, and VDC $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$2, showing substantial degradation under perturbations. The RDR-LLaVA-13B system has Clean Accuracy $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$3, PRS $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$4, and VDC $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$5, indicating that the AMCV-based framework improves robustness and answer consistency at inference time without requiring retraining or fine-tuning (Zhou et al., 23 Aug 2025).

In geometry, GDP-4B-RL achieves state-of-the-art parsing performance with $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$6 aggregate parsing accuracy on PGDP-2K and $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$7 on SGDP-1K. The paper also reports that parsed formal descriptions act as a critical cognitive scaffold for downstream reasoning: when Qwen3-VL-8B is augmented with the parsed formal language, accuracy increases by $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$8 on Geometry3K, $$\varphi(v_1, ..., v_m) = Q_k x_k ... Q_1 x_1 \; \psi(v_1, ..., v_m, x_1, ..., x_k),$$9 on PGPS9K, and $D_0$0 on SolidGeo (Wang et al., 13 Apr 2026).

In reaction diagram parsing, MACReD reaches F1 scores of $D_0$1 and $D_0$2 under hard and soft match criteria on the RxnScribe benchmark, outperforming the RxnScribe baseline, which obtains $D_0$3 and $D_0$4. The paper emphasizes robustness across diverse diagram layouts, including multi-step and tree-structured reactions (Tang et al., 27 May 2026).

The software architecture results are less optimistic. SADU reports that the best-performing model, gemini-3-flash-preview, achieves only $D_0$5 overall accuracy, while gpt-4o-mini achieves $D_0$6. Accuracy declines with diagram complexity, and hard-set patterns involving long arrows, multi arrows, non-standard orientations, and overlapping relations remain especially brittle (Ouyang et al., 5 Apr 2026).

Evidence grounding is currently the weakest axis. DRAGON reports that F1 scores are low overall, with best models reaching $D_0$7 F1 on most diagram types despite often producing correct answers. On MapIQ, Claude Opus 4.6 achieves up to $D_0$8 F1, but the broader result is a large gap between coarse localization of an answer-bearing region and full recovery of all evidence required for reasoning (Iyengar et al., 28 Apr 2026).

The earlier diagram understanding literature remains relevant as a baseline. DGGN achieves $D_0$9 mAP on AI2D, versus $Q$0 for a vanilla GRU baseline and $Q$1 for a non-RNN baseline; Dqa-Net reaches $Q$2 on AI2D QA, outperforming prior best systems (Kim et al., 2017). DSDP-Net achieves a JIG score of $Q$3, while DQA-Net reaches $Q$4 test accuracy, exceeding image-only VQA baselines (Kembhavi et al., 2016). For abstract sequence-completion problems, KLSTM achieves $Q$5 on rotation, $Q$6 on counting, $Q$7 on shape scaling, $Q$8 on other type, and $Q$9 average (Ahmed et al., 2019).

6. Interpretation, misconceptions, and research directions

A common misconception is that robustness can be reduced to clean-input answer accuracy. DRAGON directly rejects this view by showing that correct answers do not guarantee that models ground their reasoning in the diagram regions that support the prediction (Iyengar et al., 28 Apr 2026). The RDR metrics make the same point from a different angle: PRS measures correctness under perturbation, while VDC measures stability relative to the clean answer regardless of correctness (Zhou et al., 23 Aug 2025).

A second misconception is that diagram reasoning failures are only failures of high-level reasoning. Several papers identify perception as the dominant bottleneck. In geometry, the bottleneck is fine-grained visual parsing (Wang et al., 13 Apr 2026); in software architecture diagrams, explicit definitions in the prompt are reported to be crucial for accuracy, while strict output rules have less impact than definitions (Ouyang et al., 5 Apr 2026). This indicates that failures often arise before the reasoning stage proper.

A third misconception is that unstructured natural language is an adequate intermediate representation for all diagram tasks. The geometry parsing results state that augmenting with formal language, as opposed to natural language, yields consistently greater gains in reasoning because formal symbolic descriptions communicate dense, unambiguous information (Wang et al., 13 Apr 2026). Similarly, DPGs, multigraphs, and QCSPs all treat structured intermediate representations as essential rather than optional (Kembhavi et al., 2016, Tang et al., 27 May 2026, Banerjee et al., 2014).

Several convergent design principles emerge. One is explicit modularization: rule selection, fact selection, and knowledge composition in FaiRR; specialized agents in MACReD; select-and-ground or verify-and-refine prompting in DRAGON (Sanyal et al., 2022, Tang et al., 27 May 2026, Iyengar et al., 28 Apr 2026). Another is controlled perturbation as an evaluation primitive, exemplified by SciDiagram-Robust and the robustness sets used in FaiRR (Zhou et al., 23 Aug 2025, Sanyal et al., 2022). A third is evidence-centric evaluation rather than answer-only scoring (Iyengar et al., 28 Apr 2026).

This suggests that future RDR systems are likely to combine diagram parsing into explicit symbolic or graph structure, robustness testing under systematic perturbations, and faithfulness checks based on localized evidence. A plausible implication from GateKD is that confidence-gated supervision may become useful when transferring robust diagram reasoning capabilities into smaller models, especially when intermediate attention patterns or rationales are noisy (Sermsri et al., 13 May 2026). Across the literature, the consistent conclusion is not that diagram reasoning is solved, but that robustness requires models to be stable under degradation, structurally grounded in the diagram, and interpretable at the level of intermediate representations as well as final answers.