DiagramEval: Graph-Based Diagram Evaluation

• DiagramEval is a framework that views SVG diagrams as directed graphs, where text elements become nodes and connections form edges.
• It uses fine-grained metrics such as node precision, recall, and F1 to evaluate concept recovery and assess diagram logic.
• The framework further assesses multi-hop relational structure via path alignment, ensuring that generated diagrams maintain intended semantic flows.

Searching arXiv for the specified papers and any directly relevant context. arxiv_search query: "(Liang et al., 29 Oct 2025) DiagramEval Evaluating LLM-Generated Diagrams via Graphs" DiagramEval is an evaluation framework for diagrams generated by LLMs in SVG form. It is defined as a fine-grained metric suite that treats a diagram as a directed graph in which text elements become nodes and logical or data-flow connections become directed edges, and it evaluates quality through two groups of metrics: Node Alignment and Path Alignment. The framework was introduced to address the limitation that standard image-based or model-based metrics report a single latent-space similarity score yet are not sensitive to missing or spurious labels, broken relationships among concepts, or irrelevant styling variation such as colors, icon design, and layout (Liang et al., 29 Oct 2025).

1. Problem setting and motivation

Diagrams play a central role in research papers for conveying ideas, but they are often complex and labor-intensive to create. The motivating claim behind DiagramEval is that, although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure. The proposed alternative is to generate demonstration diagrams directly in textual form as SVGs, thereby leveraging recent advances in LLMs (Liang et al., 29 Oct 2025).

Within that setting, the central evaluation problem is not only visual similarity but also structural and semantic fidelity. Existing model-based metrics such as CLIPScore and DreamSim fuse the whole image into a latent space and report a single similarity score. According to the DiagramEval formulation, such metrics are neither sensitive to missing or spurious labels nor to broken relationships among concepts, and they tend to be influenced by styling choices that are not central to diagram logic. DiagramEval therefore decomposes evaluation into node-level and path-level agreement, each measured with precision, recall, and $F_1$, producing both a quantitative score and an interpretable error decomposition consisting of missing nodes, extra nodes, missing paths, and spurious paths (Liang et al., 29 Oct 2025).

This design positions DiagramEval as an evaluation method for logical structure rather than as a generic image-similarity score. A plausible implication is that the framework is particularly suited to diagrams whose communicative content depends on concept identity and relation topology rather than on photorealistic appearance.

2. Graph construction from SVG

DiagramEval constructs a text-attributed graph $G(V,E)$ from an SVG code block in two stages: node extraction and edge extraction (Liang et al., 29 Oct 2025).

The node-extraction stage begins with a draft grouping pass over every <text> item in the SVG. For each item, the pipeline parses its $(x,y)$ position, font size, and character spans. Two text items are merged if their $y$-coordinates differ by less than $K \cdot \text{font\_size}$ and their $x$-ranges overlap by at least $\tau$. In the reported evaluation setting, $K=1.5$ and $\tau=0.2$ were used for spatial grouping (Liang et al., 29 Oct 2025).

A semantic-refinement stage then applies a small vision-capable LLM. This stage merges spatially adjacent text fragments that form one concept, adds missing iconic or non-textual nodes such as a neural-net icon labeled “CNN,” and removes spurious fragments such as pure numbers and non-English labels. The resulting node set $V=\{v_i\}$ assigns each node a unique ID and a cleaned text label (Liang et al., 29 Oct 2025).

Edge extraction is handled separately. A second vision-capable LLM is shown the rendered diagram together with the node list, and it identifies every directed connection, whether represented as an arrow or a line, as a pair $G(V,E)$0. The edge set then satisfies $G(V,E)$1 (Liang et al., 29 Oct 2025).

The graph abstraction is deliberately asymmetric in emphasis: text-bearing concepts are treated as the primary atomic units, while relations are treated as directed connections among those units. This makes DiagramEval especially sensitive to whether a generated diagram preserves conceptual entities and their dependency structure.

3. Node Alignment and Path Alignment

Let $G(V,E)$2 denote the generated graph and $G(V,E)$3 the reference graph. DiagramEval defines node matching through a set

$G(V,E)$4

where $G(V,E)$5 is a text similarity, for example normalized edit distance or embedding cosine, and $G(V,E)$6 is a threshold (Liang et al., 29 Oct 2025).

From this matching set, node-level counts are defined as

$G(V,E)$7

The corresponding node metrics are

$G(V,E)$8

The paper also gives the equivalent closed form

$G(V,E)$9

Path Alignment is defined only after node alignment. DiagramEval induces matched subgraphs on the common node set

$(x,y)$0

It then considers reachability between every ordered pair of distinct matched nodes. The generated and reference path sets are

$(x,y)$1

$(x,y)$2

From these sets,

$(x,y)$3

The path-level metrics are

$(x,y)$4

or equivalently

$(x,y)$5

The operational interpretation is direct. Node Alignment measures whether the generated diagram contains the right concepts. Path Alignment measures whether those concepts participate in the same multi-hop relational structure. This suggests that Node Alignment and Path Alignment separate lexical-concept fidelity from graph-topological fidelity rather than collapsing them into a single latent score.

4. Computational procedure

The paper specifies pseudocode for both alignment stages. Node alignment performs greedy one-to-one matching: it initializes $(x,y)$6, marks all reference nodes as unmatched, and for each generated node computes similarity to every unmatched reference node. If the maximum similarity exceeds $(x,y)$7, it pairs the generated node with the best reference node, adds the pair to $(x,y)$8, and marks that reference node as matched. It then computes $(x,y)$9, $y$0, and $y$1 and returns node precision, recall, and $y$2 (Liang et al., 29 Oct 2025).

Path alignment then builds $y$3 from $y$4, induces the matched subgraphs $y$5 and $y$6, and enumerates all ordered pairs $y$7 in $y$8. If $y$9 is reachable in the generated matched subgraph, it adds $K \cdot \text{font\_size}$0 to $K \cdot \text{font\_size}$1; if the mapped counterpart pair is reachable in the reference matched subgraph, it adds $K \cdot \text{font\_size}$2 to $K \cdot \text{font\_size}$3. The algorithm finally computes $K \cdot \text{font\_size}$4, $K \cdot \text{font\_size}$5, and $K \cdot \text{font\_size}$6 and returns path precision, recall, and $K \cdot \text{font\_size}$7 (Liang et al., 29 Oct 2025).

Two design choices are notable. First, the path metric is based on reachability rather than only direct edges, so it evaluates whether a generated diagram preserves multi-hop logical flow. Second, the path comparison is conditioned on the matched node set, meaning that relation quality is assessed only after concept alignment has been established. A plausible implication is that DiagramEval discourages superficial relational credit when the underlying node identities are not correctly recovered.

5. Experimental setting and empirical findings

DiagramEval was evaluated on a dataset of 361 diagrams extracted automatically from CVPR 2025 preprints, explicitly chosen to avoid overlap with model training. The generators were Llama 4 Maverick, Gemini 2.5 Pro, and Claude 3.7 Sonnet, each uniformly prompted to produce a demonstration diagram in SVG. Nodes and edges were extracted by a lightweight Gemini-2.0 Flash model. The reported baseline metrics were CLIPScore(Text) and CLIPScore(Image) on model-generated layout captions versus reference diagrams (Liang et al., 29 Oct 2025).

Across the 361 examples, the paper reports the following representative averages for overall node and path quality.

Model	Node $K \cdot \text{font\_size}$8	Path $K \cdot \text{font\_size}$9
Llama 4 Maverick	0.35	0.20
Gemini 2.5 Pro	0.33	0.23
Claude 3.7 Sonnet	0.35	0.24

The full precision and recall breakdown further differentiates model behavior. Llama 4 Maverick achieved Node Precision $x$0, Node Recall $x$1, Path Precision $x$2, and Path Recall $x$3. Gemini 2.5 Pro achieved Node Precision $x$4, Node Recall $x$5, Path Precision $x$6, and Path Recall $x$7. Claude 3.7 Sonnet achieved Node Precision $x$8, Node Recall $x$9, Path Precision $\tau$0, and Path Recall $\tau$1 (Liang et al., 29 Oct 2025).

The accompanying observations are explicit. All three LLMs miss or mis-recognize roughly half of the reference nodes and edges. Claude 3.7 Sonnet excels at recall and tends to over-generate many nodes, which explains its high node recall and low node precision. The paper also states that CLIPScore(Image) favors visual layout over textual and logical fidelity, while CLIPScore(Text) is overly penalized by unexpected labels (Liang et al., 29 Oct 2025).

Human correlation results were reported on 50 sampled diagrams using Gemini outputs. Node $\tau$2 achieved Pearson correlation $\tau$3 with expert score, Path $\tau$4 achieved $\tau$5, CLIP(Text) achieved $\tau$6, and CLIP(Image) achieved $\tau$7.

Metric	Pearson w.r.t. expert score
Node $\tau$8	0.43
Path $\tau$9	0.41
CLIP(Text)	0.11
CLIP(Image)	0.08

The stated conclusion is that Node $K=1.5$0 and Path $K=1.5$1 correlate far better with human judgments of “same logic” than either CLIPScore variant. In the narrow sense defined by the study, DiagramEval therefore functions as a logic-sensitive evaluation protocol rather than a style-sensitive one (Liang et al., 29 Oct 2025).

6. Explainability, diagnostic cases, and relation to broader benchmarks

A central property of DiagramEval is explainability. Because it reports exactly which nodes and which multi-hop relations are missing or spurious, it can localize failure modes in generated diagrams. The paper gives four qualitative cases that illustrate how its graph-based diagnostics diverge from CLIP-based scores (Liang et al., 29 Oct 2025).

In Case A, CLIP(Text) was low at $K=1.5$2 but Path $K=1.5$3 was $K=1.5$4: the generator reproduced the entire data-flow graph perfectly but used a very different layout and iconography from the reference. In Case B, CLIP(Text) was $K=1.5$5 but Path $K=1.5$6 was $K=1.5$7: the generator printed all labels mentioned in the prompt in the correct style but failed to draw any arrows. In Case C, CLIP(Image) was low at $K=1.5$8 but Node $K=1.5$9 was $\tau=0.2$0: non-textual style differences such as background and colors lowered image similarity even though nearly all nodes were correct. In Case D, CLIP(Image) was high at $\tau=0.2$1 but Node $\tau=0.2$2 was $\tau=0.2$3: superficial layout resemblance raised the image score even though almost all concept labels were wrong or missing (Liang et al., 29 Oct 2025).

These cases formalize a recurrent misconception in diagram evaluation: that high image similarity or caption similarity necessarily implies high semantic fidelity. DiagramEval explicitly rejects that equivalence by separating concept recovery from relational recovery.

The broader diagram-as-code literature situates this contribution within a larger transition from pixel-based synthesis to symbolic representations. VCG-Bench defines a diagram-as-code paradigm using mxGraph XML, covering Vision-to-Code generation and Code-to-Code editing, and evaluates generated diagrams with metrics including mxGraph Execution Success Rate, Style Consistency Score, SigLIP2 similarity, CodeXQA, and XDRFR (Su et al., 15 May 2026). Unlike DiagramEval’s SVG-to-graph evaluation, VCG-Bench emphasizes executability, visual fidelity, and semantic compliance over mxGraph programs. This suggests a complementary division of labor: DiagramEval targets graph-structured logical fidelity in LLM-generated demonstration diagrams, whereas VCG-Bench targets structured generation and editing under a diagram-as-code framework with explicit XML validity and editability constraints (Liang et al., 29 Oct 2025, Su et al., 15 May 2026).

The VCG-Bench results also reinforce the importance of structured representations. Its reported findings indicate that Vision-to-Code remains the chief bottleneck, that current VLMs struggle with exact instance counts and full topology from pixels, and that Code-to-Code editing with incremental patches is substantially easier and highly robust (Su et al., 15 May 2026). A plausible implication is that DiagramEval’s node and path metrics could be particularly informative in regimes where topology preservation, rather than merely renderability or style consistency, is the primary concern.

Taken together, DiagramEval defines an evaluation vocabulary for diagram generation centered on graph correspondence: whether the right nodes were produced, whether the right paths exist among them, and which specific atomic elements or relations failed. In that sense, it frames diagram quality as a problem of explicit structural alignment rather than latent visual resemblance (Liang et al., 29 Oct 2025).