---
title: 'DiagramEval: Graph-Based Diagram Evaluation'
url: https://www.emergentmind.com/topics/diagrameval
type: topic
---

# DiagramEval: Graph-Based Diagram Evaluation

Searching arXiv for the specified papers and any directly relevant context.
arxiv_search query: "2510.25761 DiagramEval Evaluating LLM-Generated Diagrams via Graphs"
DiagramEval is an evaluation framework for diagrams generated by large language models in SVG form. It is defined as a fine-grained metric suite that treats a diagram as a directed graph in which text elements become nodes and logical or data-flow connections become directed edges, and it evaluates quality through two groups of metrics: Node Alignment and Path Alignment. The framework was introduced to address the limitation that standard image-based or model-based metrics report a single latent-space similarity score yet are not sensitive to missing or spurious labels, broken relationships among concepts, or irrelevant styling variation such as colors, icon design, and layout [2510.25761].

## 1. Problem setting and motivation

Diagrams play a central role in research papers for conveying ideas, but they are often complex and labor-intensive to create. The motivating claim behind DiagramEval is that, although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure. The proposed alternative is to generate demonstration diagrams directly in textual form as SVGs, thereby leveraging recent advances in LLMs [2510.25761].

Within that setting, the central evaluation problem is not only visual similarity but also structural and semantic fidelity. Existing model-based metrics such as CLIPScore and DreamSim fuse the whole image into a latent space and report a single similarity score. According to the DiagramEval formulation, such metrics are neither sensitive to missing or spurious labels nor to broken relationships among concepts, and they tend to be influenced by styling choices that are not central to diagram logic. DiagramEval therefore decomposes evaluation into node-level and path-level agreement, each measured with precision, recall, and $F_1$, producing both a quantitative score and an interpretable error decomposition consisting of missing nodes, extra nodes, missing paths, and spurious paths [2510.25761].

This design positions DiagramEval as an evaluation method for logical structure rather than as a generic image-similarity score. A plausible implication is that the framework is particularly suited to diagrams whose communicative content depends on concept identity and relation topology rather than on photorealistic appearance.

## 2. Graph construction from SVG

DiagramEval constructs a text-attributed graph $G(V,E)$ from an SVG code block in two stages: node extraction and edge extraction [2510.25761].

The node-extraction stage begins with a draft grouping pass over every `<text>` item in the SVG. For each item, the pipeline parses its $(x,y)$ position, font size, and character spans. Two text items are merged if their $y$-coordinates differ by less than $K \cdot \text{font\_size}$ and their $x$-ranges overlap by at least $\tau$. In the reported evaluation setting, $K=1.5$ and $\tau=0.2$ were used for spatial grouping [2510.25761].

A semantic-refinement stage then applies a small vision-capable LLM. This stage merges spatially adjacent text fragments that form one concept, adds missing iconic or non-textual nodes such as a neural-net icon labeled “CNN,” and removes spurious fragments such as pure numbers and non-English labels. The resulting node set $V=\{v_i\}$ assigns each node a unique ID and a cleaned text label [2510.25761].

Edge extraction is handled separately. A second vision-capable LLM is shown the rendered diagram together with the node list, and it identifies every directed connection, whether represented as an arrow or a line, as a pair $(\text{source\_ID}, \text{target\_ID})$. The edge set then satisfies $E \subseteq V \times V$ [2510.25761].

The graph abstraction is deliberately asymmetric in emphasis: text-bearing concepts are treated as the primary atomic units, while relations are treated as directed connections among those units. This makes DiagramEval especially sensitive to whether a generated diagram preserves conceptual entities and their dependency structure.

## 3. Node Alignment and Path Alignment

Let $G_{\text{gen}}=(V_{\text{gen}},E_{\text{gen}})$ denote the generated graph and $G_{\text{ref}}=(V_{\text{ref}},E_{\text{ref}})$ the reference graph. DiagramEval defines node matching through a set

$$
M_V = \{ (v_{\text{gen}},v_{\text{ref}}) \in V_{\text{gen}} \times V_{\text{ref}} \mid \text{sim}(\text{text}(v_{\text{gen}}),\text{text}(v_{\text{ref}})) \ge \tau_{\text{text}} \},
$$

where $\text{sim}(\cdot,\cdot)$ is a text similarity, for example normalized edit distance or embedding cosine, and $\tau_{\text{text}}$ is a threshold [2510.25761].

From this matching set, node-level counts are defined as

$$
TP_V = |M_V|,\qquad
FP_V = |V_{\text{gen}}| - TP_V,\qquad
FN_V = |V_{\text{ref}}| - TP_V.
$$

The corresponding node metrics are

$$
\text{Precision}_V = \frac{TP_V}{TP_V + FP_V},\qquad
\text{Recall}_V = \frac{TP_V}{TP_V + FN_V},\qquad
F1_V = \frac{2 \cdot \text{Precision}_V \cdot \text{Recall}_V}{\text{Precision}_V+\text{Recall}_V}.
$$

The paper also gives the equivalent closed form

$$
\text{Precision}_V = \frac{|M_V|}{|V_{\text{gen}}|},\qquad
\text{Recall}_V = \frac{|M_V|}{|V_{\text{ref}}|},\qquad
F1_V = \frac{2\,|M_V|}{|V_{\text{gen}}| + |V_{\text{ref}}|}.
$$

Path Alignment is defined only after node alignment. DiagramEval induces matched subgraphs on the common node set

$$
V_M = \{v_{\text{gen}} \mid \exists v_{\text{ref}}:(v_{\text{gen}},v_{\text{ref}})\in M_V\}
     = \{v_{\text{ref}} \mid \exists v_{\text{gen}}:(v_{\text{gen}},v_{\text{ref}})\in M_V\}.
$$

It then considers reachability between every ordered pair of distinct matched nodes. The generated and reference path sets are

$$
P_{\text{gen}} = \{ (u,v) \mid u \ne v,\ \text{path from } u \to v \text{ exists in } G_{\text{gen}}^M \},
$$

$$
P_{\text{ref}} = \{ (u,v) \mid u \ne v,\ \text{path from } u' \to v' \text{ exists in } G_{\text{ref}}^M,\ \text{where } u',v' \text{ are the } M_V\text{-mapped counterparts of } u,v \}.
$$

From these sets,

$$
TP_P = |P_{\text{gen}} \cap P_{\text{ref}}|,\qquad
FP_P = |P_{\text{gen}} \setminus P_{\text{ref}}|,\qquad
FN_P = |P_{\text{ref}} \setminus P_{\text{gen}}|.
$$

The path-level metrics are

$$
\text{Precision}_P = \frac{TP_P}{TP_P + FP_P},\qquad
\text{Recall}_P = \frac{TP_P}{TP_P + FN_P},\qquad
F1_P = \frac{2 \cdot \text{Precision}_P \cdot \text{Recall}_P}{\text{Precision}_P+\text{Recall}_P},
$$

or equivalently

$$
\text{Precision}_P = \frac{|P_{\text{gen}} \cap P_{\text{ref}}|}{|P_{\text{gen}}|},\qquad
\text{Recall}_P = \frac{|P_{\text{gen}} \cap P_{\text{ref}}|}{|P_{\text{ref}}|},\qquad
F1_P = \frac{2\,|P_{\text{gen}} \cap P_{\text{ref}}|}{|P_{\text{gen}}| + |P_{\text{ref}}|}.
$$

The operational interpretation is direct. Node Alignment measures whether the generated diagram contains the right concepts. Path Alignment measures whether those concepts participate in the same multi-hop relational structure. This suggests that Node Alignment and Path Alignment separate lexical-concept fidelity from graph-topological fidelity rather than collapsing them into a single latent score.

## 4. Computational procedure

The paper specifies pseudocode for both alignment stages. Node alignment performs greedy one-to-one matching: it initializes $M_V \leftarrow \varnothing$, marks all reference nodes as unmatched, and for each generated node computes similarity to every unmatched reference node. If the maximum similarity exceeds $\tau_{\text{text}}$, it pairs the generated node with the best reference node, adds the pair to $M_V$, and marks that reference node as matched. It then computes $TP_V$, $FP_V$, and $FN_V$ and returns node precision, recall, and $F_1$ [2510.25761].

Path alignment then builds $V_M$ from $M_V$, induces the matched subgraphs $G_{\text{gen}}^M$ and $G_{\text{ref}}^M$, and enumerates all ordered pairs $u \ne v$ in $V_M$. If $u \to v$ is reachable in the generated matched subgraph, it adds $(u,v)$ to $P_{\text{gen}}$; if the mapped counterpart pair is reachable in the reference matched subgraph, it adds $(u,v)$ to $P_{\text{ref}}$. The algorithm finally computes $TP_P$, $FP_P$, and $FN_P$ and returns path precision, recall, and $F_1$ [2510.25761].

Two design choices are notable. First, the path metric is based on reachability rather than only direct edges, so it evaluates whether a generated diagram preserves multi-hop logical flow. Second, the path comparison is conditioned on the matched node set, meaning that relation quality is assessed only after concept alignment has been established. A plausible implication is that DiagramEval discourages superficial relational credit when the underlying node identities are not correctly recovered.

## 5. Experimental setting and empirical findings

DiagramEval was evaluated on a dataset of 361 diagrams extracted automatically from CVPR 2025 preprints, explicitly chosen to avoid overlap with model training. The generators were Llama 4 Maverick, Gemini 2.5 Pro, and Claude 3.7 Sonnet, each uniformly prompted to produce a demonstration diagram in SVG. Nodes and edges were extracted by a lightweight Gemini-2.0 Flash model. The reported baseline metrics were CLIPScore(Text) and CLIPScore(Image) on model-generated layout captions versus reference diagrams [2510.25761].

Across the 361 examples, the paper reports the following representative averages for overall node and path quality.

| Model | Node $F_1$ | Path $F_1$ |
|---|---:|---:|
| Llama 4 Maverick | 0.35 | 0.20 |
| Gemini 2.5 Pro | 0.33 | 0.23 |
| Claude 3.7 Sonnet | 0.35 | 0.24 |

The full precision and recall breakdown further differentiates model behavior. Llama 4 Maverick achieved Node Precision $0.47$, Node Recall $0.31$, Path Precision $0.23$, and Path Recall $0.25$. Gemini 2.5 Pro achieved Node Precision $0.36$, Node Recall $0.37$, Path Precision $0.25$, and Path Recall $0.28$. Claude 3.7 Sonnet achieved Node Precision $0.29$, Node Recall $0.51$, Path Precision $0.34$, and Path Recall $0.21$ [2510.25761].

The accompanying observations are explicit. All three LLMs miss or mis-recognize roughly half of the reference nodes and edges. Claude 3.7 Sonnet excels at recall and tends to over-generate many nodes, which explains its high node recall and low node precision. The paper also states that CLIPScore(Image) favors visual layout over textual and logical fidelity, while CLIPScore(Text) is overly penalized by unexpected labels [2510.25761].

Human correlation results were reported on 50 sampled diagrams using Gemini outputs. Node $F_1$ achieved Pearson correlation $0.43$ with expert score, Path $F_1$ achieved $0.41$, CLIP(Text) achieved $0.11$, and CLIP(Image) achieved $0.08$.

| Metric | Pearson w.r.t. expert score |
|---|---:|
| Node $F_1$ | 0.43 |
| Path $F_1$ | 0.41 |
| CLIP(Text) | 0.11 |
| CLIP(Image) | 0.08 |

The stated conclusion is that Node $F_1$ and Path $F_1$ correlate far better with human judgments of “same logic” than either CLIPScore variant. In the narrow sense defined by the study, DiagramEval therefore functions as a logic-sensitive evaluation protocol rather than a style-sensitive one [2510.25761].

## 6. Explainability, diagnostic cases, and relation to broader benchmarks

A central property of DiagramEval is explainability. Because it reports exactly which nodes and which multi-hop relations are missing or spurious, it can localize failure modes in generated diagrams. The paper gives four qualitative cases that illustrate how its graph-based diagnostics diverge from CLIP-based scores [2510.25761].

In **Case A**, CLIP(Text) was low at $0.26$ but Path $F_1$ was $1.0$: the generator reproduced the entire data-flow graph perfectly but used a very different layout and iconography from the reference. In **Case B**, CLIP(Text) was $1.0$ but Path $F_1$ was $0.0$: the generator printed all labels mentioned in the prompt in the correct style but failed to draw any arrows. In **Case C**, CLIP(Image) was low at $0.60$ but Node $F_1$ was $0.87$: non-textual style differences such as background and colors lowered image similarity even though nearly all nodes were correct. In **Case D**, CLIP(Image) was high at $0.81$ but Node $F_1$ was $0.16$: superficial layout resemblance raised the image score even though almost all concept labels were wrong or missing [2510.25761].

These cases formalize a recurrent misconception in diagram evaluation: that high image similarity or caption similarity necessarily implies high semantic fidelity. DiagramEval explicitly rejects that equivalence by separating concept recovery from relational recovery.

The broader diagram-as-code literature situates this contribution within a larger transition from pixel-based synthesis to symbolic representations. VCG-Bench defines a diagram-as-code paradigm using mxGraph XML, covering Vision-to-Code generation and Code-to-Code editing, and evaluates generated diagrams with metrics including mxGraph Execution Success Rate, Style Consistency Score, SigLIP2 similarity, CodeXQA, and XDRFR [2605.15677]. Unlike DiagramEval’s SVG-to-graph evaluation, VCG-Bench emphasizes executability, visual fidelity, and semantic compliance over mxGraph programs. This suggests a complementary division of labor: DiagramEval targets graph-structured logical fidelity in LLM-generated demonstration diagrams, whereas VCG-Bench targets structured generation and editing under a diagram-as-code framework with explicit XML validity and editability constraints [2510.25761] [2605.15677].

The VCG-Bench results also reinforce the importance of structured representations. Its reported findings indicate that Vision-to-Code remains the chief bottleneck, that current VLMs struggle with exact instance counts and full topology from pixels, and that Code-to-Code editing with incremental patches is substantially easier and highly robust [2605.15677]. A plausible implication is that DiagramEval’s node and path metrics could be particularly informative in regimes where topology preservation, rather than merely renderability or style consistency, is the primary concern.

Taken together, DiagramEval defines an evaluation vocabulary for diagram generation centered on graph correspondence: whether the right nodes were produced, whether the right paths exist among them, and which specific atomic elements or relations failed. In that sense, it frames diagram quality as a problem of explicit structural alignment rather than latent visual resemblance [2510.25761].

Source: https://www.emergentmind.com/topics/diagrameval