DiagramIR: Scalable Diagram Evaluation

• DiagramIR is a scalable evaluation pipeline that converts TikZ code into an intermediate structured representation for automated, deterministic diagram checking.
• It employs a three-stage process—back-translation, normalization, and rule-based evaluation—that achieves high agreement with human raters at a fraction of the cost.
• The framework enables smaller language models to perform competitively with larger ones by leveraging structured geometric analysis for cost-effective educational applications.

Searching arXiv for the target paper and closely related diagram-representation work. Searching arXiv for "(Kumar et al., 11 Nov 2025) DiagramIR" and related titles. DiagramIR is an automatic and scalable evaluation pipeline for geometric figures motivated by the use of LLMs in learning environments where visualizations are essential, such as mathematics. In this usage, DiagramIR denotes a method that evaluates educational diagrams by operating on intermediate representations of LaTeX TikZ code rather than treating the task as unconstrained free-form judgment. The pipeline compares favorably with baselines such as LLM-as-a-Judge, has higher agreement with human raters, and enables smaller models such as GPT-4.1-Mini to perform comparably to larger models such as GPT-5 at a 10x lower inference cost, a property presented as important for accessible and scalable education technologies (Kumar et al., 11 Nov 2025).

1. Problem setting and objective

DiagramIR is situated in a setting where LLMs can generate code that compiles to educational figures, but scalable evaluation remains a bottleneck. The central design choice is to make diagram evaluation automatic and scalable by introducing an intermediate representation for geometric figures expressed in TikZ. This representation serves as the basis for downstream checking, rather than relying exclusively on direct model judgments over code, rendered images, or both (Kumar et al., 11 Nov 2025).

The problem formulation is specific. The input is raw TikZ code; the output is an evaluation report over rubric criteria. The pipeline is described as having three major stages: TikZ to IR back-translation by an LLM, normalization and data-structure construction, and deterministic rule-based checks. This organization places DiagramIR at the intersection of program understanding, geometric representation, and rubric-based educational assessment (Kumar et al., 11 Nov 2025).

A recurrent misconception is that the system is merely an LLM-as-a-Judge prompt template specialized to diagrams. The published description instead defines a hybrid architecture in which the LLM is used for back-translation into a schema-constrained pivot language, while the actual evaluation proceeds through deterministic geometry and spatial checks. This distinction is central to the method’s reported agreement and cost profile (Kumar et al., 11 Nov 2025).

2. Intermediate representation and formal schema

The intermediate representation is a JSON-style pivot language that captures geometric primitives and text. Its core top-level fields are "canvas", "nodes", "shapes", "segments", "arcs", and "texts". At the object level, nodes carry identifiers and coordinates; shapes have a "type" such as "polygon", "circle", or "rectangle" together with point references; segments connect two node identifiers; arcs specify center, radius, and angular bounds; and texts store content, position, and style (Kumar et al., 11 Nov 2025).

This schema is illustrated with a right-triangle example whose TikZ source declares coordinates A, B, and C, draws three segments, and places the labels 3, 4, and 5. In the corresponding IR, the triangle appears as a polygon over the points ["A","B","C"], the edges appear in the segment list, and the side labels appear in the text list with explicit positions. The example clarifies that DiagramIR does not preserve TikZ as opaque source code; it converts it into a structured geometric object language suitable for algorithmic analysis (Kumar et al., 11 Nov 2025).

The representation is paired with in-memory structures used during evaluation. These include a point-graph $G_p=(V_p,E_p)$, where $V_p$ contains all nodes and text anchors and $E_p$ contains segment endpoints and polygon edges; a shape-list $S$ for polygons, circles, and rectangles; an arc-list $A$; a text-list $T$; and a relationship graph $R$. In $R$, nodes range over points, segments, arcs, shapes, and texts, while edges are labeled by predicates such as incidence(p, segment), adjacency(segment, segment), parallelism(seg_i, seg_j), and containment(text, shape) (Kumar et al., 11 Nov 2025).

The IR can also be normalized geometrically. The published normalization translates and scales the diagram so that its bounding box fits $[0,1]\times[0,1]$, with

$$x'=\frac{x-x_{\min}}{\max(x_{\max}-x_{\min},\,y_{\max}-y_{\min})},\qquad y'=\frac{y-y_{\min}}{\max(x_{\max}-x_{\min},\,y_{\max}-y_{\min})}.$$

This makes subsequent geometric checks less sensitive to absolute coordinate scale and suggests a canonicalization step for cross-diagram comparison (Kumar et al., 11 Nov 2025).

3. Pipeline architecture and execution model

The execution flow is explicit. A high-level procedure begins with raw TikZ code, formats a prompt, calls an LLM to obtain IR JSON, parses that JSON, optionally normalizes coordinates, runs the check modules, aggregates pass/fail outcomes for each rubric criterion, and returns an evaluation report. In pseudocode form, the core function is EvaluateDiagram(tikz_code), with the sequence formatPrompt, LLM.call, parseJSON, normalizeCoordinates, and runChecks (Kumar et al., 11 Nov 2025).

At the architectural level, DiagramIR is therefore neither a pure symbolic parser nor a pure neural evaluator. The LLM is responsible for the TikZ-to-IR back-translation step, after which the system transitions into graph construction and deterministic analysis. The relationship graph then supports checks over geometry and layout, using the labeled relations already encoded during preprocessing. This design localizes ambiguity to the back-translation phase and reserves explicit rule execution for the evaluation phase (Kumar et al., 11 Nov 2025).

The check stage is described as geometry plus spatial evaluation. While the full rule inventory is not enumerated in the summary, the available predicates make clear that the intended checks cover incidence, adjacency, parallelism, and containment. A plausible implication is that rubric criteria can be implemented as predicates over the IR and its derived graphs rather than as free-form natural-language grading, which is consistent with the system’s pass/fail aggregation strategy (Kumar et al., 11 Nov 2025).

4. Evaluation methodology and comparison framework

The main evaluation is rubric-based, with pass/fail outcomes for each criterion. The summary additionally defines several comparison metrics that can be used for diagram-to-reference assessment: precision,

$V_p$0

recall,

$V_p$1

and

$V_p$2

For spatial overlap it gives

$V_p$3

These formulas are presented as available evaluative machinery even though the principal reported setup remains rubric-based (Kumar et al., 11 Nov 2025).

Structural and semantic comparison can also be cast as attributed graph comparison via graph-edit distance. The summary defines GED as the minimal cost of insertions, deletions, and substitutions of nodes and edges needed to align the graph of a generated diagram with that of a reference diagram. A zero GED corresponds to isomorphic structures. This framing is technically significant because the IR already induces graph structure, making graph-level comparison a natural extension of the basic rule-based pipeline (Kumar et al., 11 Nov 2025).

The principal baseline is LLM-as-a-Judge. In that configuration, the same or a larger LLM is prompted with a rubric and receives either only TikZ code, only the rendered image, or both code and image, and is asked to output pass/fail decisions with rationale. DiagramIR differs by inserting a formal representation and deterministic checks between the model and the final grading decision. This architectural difference underlies the direct comparison reported in the experimental section (Kumar et al., 11 Nov 2025).

5. Reported results, cost profile, and deployment

The reported quantitative comparison centers on back-translation (BT) versus LLM-as-a-Judge. Cohen’s $V_p$4 is used to measure agreement with human raters. The selected results given in the paper are as follows (Kumar et al., 11 Nov 2025):

Model	BT: $V_p$5, time, cost	Judge: $V_p$6, time, cost
GPT-5	$V_p$7, $V_p$8s, $V_p$910.29$E_p$00.498$E_p$126.20$E_p$2\$E_p$3
GPT-4.1-Mini	$E_p$4, $E_p$5s, $E_p$60.47$E_p$70.388$E_p$88.55$E_p$9\$S$0

These rows support two distinct claims made in the summary. First, back-translation achieves higher or comparable $S$1 than the judge setup. Second, smaller models become economically attractive within the DiagramIR pipeline: the average cost per diagram is reported as $S$20.47/386 \approx \$S$3 for BT with GPT-4.1-Mini, compared with $S$44.85/386 \approx \$S$5 for LLM-as-Judge using GPT-5, which is summarized as roughly 10× lower cost for the full evaluation pipeline (Kumar et al., 11 Nov 2025).

Deployment is framed in microservice terms. The proposed interface is a REST API with a POST /evaluate endpoint that accepts TikZ code and returns rubric results together with the IR. The surrounding web-based tutoring workflow is straightforward: students or teachers submit diagram requests, a generator returns TikZ, DiagramIR is called immediately, and the frontend displays green/red checks inline. Caching identical code submissions, containerizing the IR-parser and rule-check modules, and autoscaling based on request volume are all identified as deployment considerations. The summary also names GPT-4.1-Mini, or a fine-tuned small transformer, as a low-cost option for TikZ-to-IR parsing (Kumar et al., 11 Nov 2025).

6. Relation to other diagram-oriented intermediate representations

The name DiagramIR in (Kumar et al., 11 Nov 2025) refers specifically to educational math diagram evaluation through intermediate representations of TikZ code. The broader literature, however, uses closely related language for substantially different technical objects.

One adjacent line of work treats diagrams as multimodal inputs from which a relational graph should be inferred. The unified diagram-parsing network UDPnet and its Dynamic Graph Generation Network begin with diagram object detection, score ordered object pairs for edge existence, maintain a dynamic adjacency tensor memory, and emit a structured graph for downstream retrieval or question answering (Kim et al., 2017). This is a graph-generation problem rather than a rubric-based evaluator, but it shares the idea that diagram understanding benefits from an explicit intermediate relational structure.

A second line of work uses a Diagram IR to specify compositional training regimes. In “DiagrammaticLearning,” a learning diagram is a labeled graph with a preorder on edges; it is interpreted as a functor

$S$6

and compiles to a unique loss function obtained by summing losses over parallel pairs of paths. The accompanying library supports PyTorch and Flux.jl models, and examples include knowledge distillation, few-shot multi-task learning, and multimodal learning (Lary et al., 2 Jan 2025). Here the IR is not a representation of a visual figure, but of a training setup.

A third usage appears in information visualization for component-based evaluation in information retrieval systems. The tool described in “An InfoVis Tool for Interactive Component-Based Evaluation” is a fully interactive Sankey diagram over a Grid of Points containing 612 systems, six TREC collections, 50 topics each, and eight effectiveness measures. It computes marginal means and lightweight multiple-comparison tests in real time to support visual exploration of IR component combinations (Rocco et al., 2019). Despite the name overlap, this DiagramIR is an InfoVis environment for search-system evaluation rather than a diagram-evaluation pipeline.

This suggests that “DiagramIR” and “Diagram IR” function as a family of terms for explicit, machine-operable representations of diagrammatic or graph-structured objects, but the semantics differ sharply across subfields. In educational geometry evaluation, the term denotes a TikZ-centered pivot language plus deterministic checks; in diagram parsing, it points toward recovered relational graphs; in compositional ML, it denotes a graph-based specification of losses; and in information retrieval evaluation, it names a Sankey-based analytical interface (Kumar et al., 11 Nov 2025, Kim et al., 2017, Lary et al., 2 Jan 2025, Rocco et al., 2019).