VisuGen Metrics: Evaluating Visual Generation

• VisuGen Metrics are a family of evaluation frameworks that decompose visual generation quality into distinct aspects such as visual grounding, creativity, and instruction adherence.
• They integrate reference-based, distribution-aware, and structural metrics to assess diverse outputs in tasks like image captioning, chart generation, and diagrammatic conversion.
• Empirical results demonstrate that these metrics reliably capture fidelity, diversity, and compliance, thereby guiding improvements in vision-language generation models.

Searching arXiv for papers explicitly using or defining “VisuGen Metrics” and closely related visualization-generation evaluation metrics. In the cited literature, “VisuGen Metrics” denotes a cluster of evaluation frameworks for generation tasks conditioned on visual inputs or producing visual artifacts. The term is used most explicitly in VisuCraft for the triplet Visual Grounding (VG), Creativity (C), and Instruction Adherence (IA), averaged into a Mean score for long-form creative text generation from images (Jiang et al., 4 Aug 2025). Closely related work extends the same evaluative agenda to distribution-aware caption assessment, supervised calibration of caption metrics against human judgments, chart-generation scoring at the specification and rendered-image levels, structural analysis of SVG outputs, and component-wise accuracy measurement for Image→PlantUML conversion (Chan et al., 2022, Winata et al., 2024, Hostnik et al., 21 Jan 2026, Zhu et al., 9 Apr 2026, Ranjani et al., 15 Sep 2025). Taken together, these frameworks treat evaluation not as a single scalar proxy, but as a decomposition over fidelity, diversity, compliance, structure, and human alignment.

1. Terminological scope and evaluative targets

The most direct formalization of VisuGen Metrics appears in VisuCraft, where the metric family is defined as Visual Grounding, Creativity, and Instruction Adherence, with a simple arithmetic Mean over the three axes (Jiang et al., 4 Aug 2025). In other papers, the same label or a closely related “VisuGen” framing is applied to different output spaces: SVG generation via leave-one-out structural analysis, Image→PlantUML conversion via insertion/deletion/substitution rates, and visual-description evaluation via distribution-aware comparison of caption sets rather than single captions (Zhu et al., 9 Apr 2026, Ranjani et al., 15 Sep 2025, Chan et al., 2022).

These frameworks share a common motivation. Traditional evaluation pipelines often reduce performance to pairwise overlap against references or to a single similarity score, which can miss semantically valid diversity, structural modularity, editability, or prompt compliance. Chan et al. explicitly argue that existing metrics are not appropriate for domains such as visual description where reference diversity is signal rather than noise (Chan et al., 2022). A separate audit of image-captioning metrics shows that different metrics measure sharply different properties: n-gram overlap, scene-graph semantics, contextual semantic similarity, and image–text compatibility do not collapse to the same notion of quality (González-Chávez et al., 2022).

This suggests that “VisuGen Metrics” is best understood not as a single standardized benchmark, but as a family of task-specific metrics designed to expose failure modes that single-score evaluation suppresses.

2. The VisuCraft triad: Visual Grounding, Creativity, and Instruction Adherence

In VisuCraft, Visual Grounding (VG) measures how faithfully the generated text $T$ realizes the structured visual attributes $V=\{v_1,\dots,v_M\}$ extracted from an input image. With $\mathrm{match}(v_i,T)=1$ if attribute $v_i$ is realized in $T$ and $0$ otherwise, the score is

$$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$

The matching procedure uses exact or fuzzy-string matching plus synonym expansion. The paper’s intuition is operational: VG measures whether the output text “mentions” or reflects the fine-grained visual attributes produced by the extractor $\mathcal{E}$, such as “desolate cliff,” “flickering lighthouse beam,” or “stormy gray sea” (Jiang et al., 4 Aug 2025).

Creativity (C) is defined as a weighted combination of lexical diversity and semantic novelty relative to the training corpus. The lexical term is

$$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$

and the novelty term is

$$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$

With default $V=\{v_1,\dots,v_M\}$0, the combined score is

$V=\{v_1,\dots,v_M\}$1

This construction fixes creativity to two explicit signals: local lexical variation and corpus-level semantic non-redundancy. It does not treat creativity as an unconstrained human-aesthetic judgment; rather, it operationalizes it through diversity and novelty (Jiang et al., 4 Aug 2025).

Instruction Adherence (IA) measures fulfillment of user-specified sub-goals. Given instruction $V=\{v_1,\dots,v_M\}$2, a parser splits it into $V=\{v_1,\dots,v_M\}$3 atomic sub-goals $V=\{v_1,\dots,v_M\}$4, and the score is

$V=\{v_1,\dots,v_M\}$5

where $V=\{v_1,\dots,v_M\}$6 is determined by a zero-shot natural-language inference model or, when appropriate, by string or sentiment-analysis checks. The aggregate metric is

$V=\{v_1,\dots,v_M\}$7

The evaluation protocol uses ImageStoryGen-500K, comprising 500 000 image–instruction pairs, with 400 k training, 50 k validation, and 50 k test examples across StoryGen, Poetry, and AdCopyGen. Final evaluation samples 5 000 test cases per scenario, for 15 000 total, using stratified sampling. Human-in-the-loop calibration includes a validation of the sub-goal parser on 500 instruction texts, with 92% end-to-end accuracy, and fuzzy-matching thresholds tuned on a held-out 5K validation subset, aligning automatic VG with human judgments at Pearson $V=\{v_1,\dots,v_M\}$8 (Jiang et al., 4 Aug 2025).

The reported results show consistent gains for VisuCraft over baselines. On StoryGen, VisuCraft scores VG 0.825 ± .002, C 0.810 ± .002, IA 0.830 ± .002, and Mean 0.822 ± .002, compared with LVLM-Enh Mean 0.811 ± .002 and LVLM-Base Mean 0.781 ± .003. On Poetry, the corresponding Means are 0.810 ± .003, 0.794 ± .003, and 0.772 ± .003. On AdCopyGen, VisuCraft reports Mean 0.803 ± .003. All reported differences versus LVLM-Enhanced are significant at $V=\{v_1,\dots,v_M\}$9, with Cohen’s $\mathrm{match}(v_i,T)=1$0 for Mean score equal to 0.43 on StoryGen, 0.38 on Poetry, and 0.35 on AdCopyGen (Jiang et al., 4 Aug 2025).

Ablations further specify what the metric family is sensitive to. Removing $\mathrm{match}(v_i,T)=1$1 or $\mathrm{match}(v_i,T)=1$2 degrades all three VisuGen Metrics, and richer structured visual granularity monotonically improves VG, C, IA, and Mean from Level 1: Object names to Level 3: Full structure. Reported correlations over all 15 000 samples are VG vs. IA = 0.82, VG vs. C = 0.47, and C vs. IA = 0.39, while failure cases include highly abstract scenes and overly long instructions with more than five sub-goals (Jiang et al., 4 Aug 2025).

3. From single-sample caption scores to distribution-aware visual-description evaluation

A major precursor to later VisuGen-style evaluation is the shift from single-caption comparison to distribution-aware metrics for visual description. Chan et al. define an image- or video-conditioned human reference distribution $\mathrm{match}(v_i,T)=1$3 and model distribution $\mathrm{match}(v_i,T)=1$4, then compare finite samples

$\mathrm{match}(v_i,T)=1$5

Instead of scoring one candidate against one or more references by max or average overlap, the framework compares the two distributions through Triangle-Rank Metrics (TRM) and Kernel-Based Metrics (KBM) (Chan et al., 2022).

For TRM, given a base distance $\mathrm{match}(v_i,T)=1$6, all directed triangles containing at least one sample from $\mathrm{match}(v_i,T)=1$7 and at least one from $\mathrm{match}(v_i,T)=1$8 are enumerated. Let $\mathrm{match}(v_i,T)=1$9 be the edge connecting two samples from the same set. If $v_i$0, $v_i$1, and $v_i$2 indicate whether $v_i$3 is the shortest, middle, or longest edge, then

$v_i$4

and

$v_i$5

Under the null hypothesis $v_i$6, each triangle edge is equally likely to be the in-distribution edge, so each term is near zero; larger $v_i$7 indicates greater divergence. KBMs include Fréchet BERT Distance (FBD) and MMD-BERT, using Sentence-BERT embeddings and either Gaussian-distribution comparison or maximum mean discrepancy (Chan et al., 2022).

The empirical point is that semantically rich domains, such as image and video description, contain genuine reference diversity. Under single-sample pairwise metrics, models are driven toward a single central caption that overlaps with all references in common n-grams while omitting rarer but valid details. Chan et al. report that under single-sample evaluation ($v_i$8) no metric can reliably distinguish $v_i$9 from $T$0 because all $T$1, whereas TRM_METEOR and TRM_CIDEr achieve significance much faster as $T$2 increases; for example, TRM_METEOR reaches $T$3 at $T$4 versus $T$5 for METEOR (Chan et al., 2022).

The framework also changes how quality–diversity trade-offs are interpreted. Standard METEOR drops monotonically as temperature increases, but TRM_METEOR peaks at moderate temperature, capturing a regime where diversity improves coverage of reference modes without excessively harming per-sample overlap. Best-practice recommendations in the paper are explicit: report distribution-aware metrics alongside single-sample scores, sample multiple captions with $T$6, prefer stochastic decoding over beam search, and use TRM on a base metric such as METEOR or CIDEr for interpretability and analytic $T$7-values (Chan et al., 2022).

A complementary critique of caption metrics on MS-COCO 2017 validation shows why these alternatives matter. In artificial “replacing” and “shuffling” scenarios, RefCLIPScore is best on replacing with Spearman $T$8, while BERTScore is best on shuffling with $T$9. The study concludes that n-gram metrics excel at penalizing random or heavy word substitution but largely fail to detect scrambled word order, whereas CLIP-based metrics capture image–text compatibility and BERTScore captures semantic or fluency degradation (González-Chávez et al., 2022). This directly supports the view that no single conventional caption metric exhausts visually grounded quality.

4. Supervised calibration and metric fusion in image captioning

MetaMetrics generalizes the idea of task-specific evaluation by learning a supervised fusion over existing metrics to maximize alignment with human judgments. Any base metric $0$0 is treated as a black-box function mapping an input $0$1 to a scalar score $0$2, and the meta-metric is defined by a fusion function $0$3 over the vector $0$4. In the simplest form,

$0$5

with weights learned to maximize correlation with human preference scores $0$6 (Winata et al., 2024).

After preprocessing by clipping, min–max normalization to $0$7, and inversion when higher means worse, the optimization target is

$0$8

where in practice $0$9 is Kendall’s $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$0 on held-out human-rated examples. Under Bayesian Optimization, a Gaussian-process prior is placed on the unknown objective $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$1, with a Matern kernel. As an alternative, the framework trains an XGBoost regressor to predict $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$2 from the same feature vector, followed by iterative pruning of the least-important metrics (Winata et al., 2024).

For image captioning, black-Cap fuses BLEU-1…4, ROUGE-L, METEOR, SPICE (F/Prec/Rec), CIDEr, TIGEr, and CLIPScore, including both reference-based and reference-free CLIPScore variants. The vision-task setup uses Flickr8k-Expert with 5 882 examples and a 30%/70% train/test split, and THumB 1.0 with 2 000 examples and the same split. Human labels are on a 1–4 scale for Flickr8k and a 1–5 rubric scale for THumB. For Bayesian Optimization, the reported hyperparameters are init_points=5 and n_iter=100; XGBoost uses n_estimators in 100,1 000 with objective=reg:squaredlogerror (Winata et al., 2024).

The reported improvements are substantial. On Flickr8k in-dataset, the best single metric is SPICE(P) ≈ 0.603, with SPICE(F) ≈ 0.591, SPICE(R) ≈ 0.579, and CLIPScore ≈ 0.514, whereas black-Cap (GP) achieves $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$3 and black-Cap (XGBoost) $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$4. On THumB 1.0, the best single metric is CLIPScore $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$5, improved to 0.329 by black-Cap (GP). In cross-dataset evaluation, the best individual metric averages about 0.403, while black-Cap (XGBoost iterative pruning) reaches 0.478, a gain of 0.075. An ablation shows that the top-5 metrics retain more than 95% of full-model performance, and feature-importance maps consistently rely on complementary signals such as SPICE, CIDEr, CLIPScore, plus METEOR or TIGEr as needed (Winata et al., 2024).

The broader significance is methodological. MetaMetrics is described as modality-agnostic and language-agnostic, with fully parallelizable inference and sparse fusion through automatic pruning. At the same time, the current implementation restricts base metrics to models of at most 10.7B parameters and avoids “massive” LLM evaluators to ensure GPU compatibility (≤48 GB). Future extensions proposed in the paper include video captioning/diffusion quality (FID/IS), audio generation, larger multilingual/cultural benchmarks, and more fine-grained aspects of human preference such as factuality vs style (Winata et al., 2024).

5. Specification-level and image-level metrics for visualization generation

For NL2VIS and chart generation, VegaChat introduces two complementary metrics: Spec Score and Vision Score. Spec Score compares a generated Vega-Lite JSON specification against a reference specification by extracting sets of mark types, encoding channels, and data transforms, then computing precision, recall, and F-scores for each component. Encodings are recall-weighted with $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$6:

$$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$7

With $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$8 for encodings and $$VG(T;V) = \frac{1}{M} \sum_{i=1}^{M} \mathrm{match}\bigl(v_i,\,T\bigr)\quad\in[0,1].$$9 for mark and transform, the final score is

$\mathcal{E}$0

where the paper chooses the encoding weight highest, for example 0.6, with 0.2 each for mark and transform, plus a small positive bonus if the spec is valid and non-empty and a heavy penalty if empty (Hostnik et al., 21 Jan 2026).

Vision Score is image-based and library-agnostic. A multimodal LLM receives the user prompt, a reference chart image, and a generated chart image, then rates five dimensions on $\mathcal{E}$1: visualization type, data encoding, data transformations, prompt compliance, and aesthetics/layout. If $\mathcal{E}$2 is the score for dimension $\mathcal{E}$3 and $\mathcal{E}$4 are manual weights summing to 1, then

$\mathcal{E}$5

The weights place highest emphasis on encoding and lowest on aesthetics. This separates semantic chart correctness from styling while explicitly incorporating prompt compliance, which many chart metrics omit (Hostnik et al., 21 Jan 2026).

The evaluation uses the NLV Corpus with 814 single-turn utterances and 59 sequential sessions, and an annotated ChartLLM subset with 48 real-world chart tasks and 144 examples total. On non-sequential NLV, VegaChat reports Spec Score 83.9%, Vision Score 85.1%, MPB 61.9%, SEVQ 90.6%, VER 0.0%, and ECR 0.3%. On ChartLLM, the corresponding values are Spec 52.6%, Vision 56.7%, MPB 26.2%, SEVQ 80.1%, VER 0.0%, and ECR 0.8%. On sequential NLV, Spec 86.5%, Vision 85.7%, and ECR 0.0% are reported. Correlation with human judgments over 171 examples is Pearson $\mathcal{E}$6 for Spec Score and $\mathcal{E}$7 for Vision Score (Hostnik et al., 21 Jan 2026).

The limitations are equally explicit. Spec Score is sensitive to structural mismatches in Vega-Lite JSON and does not handle interactive features or multi-table joins. Vision Score depends on an expensive multimodal LLM, tends to assign higher scores in ambiguous cases, and uses manually chosen weights that may not reflect all user preferences. More generally, question-type prompts can admit multiple valid visualizations, so a reference chart may differ from an equally valid generated response (Hostnik et al., 21 Jan 2026).

A useful contrast is provided by VizGen, whose evaluation is entirely based on an online Google Forms survey of $\mathcal{E}$8 participants. Reported measures are average Likert-scale ratings, such as Ease of Use 4.4/5.0, Confidence Without Support 4.4/5.0, Overall UI Design 4.4/5.0, Clarity of Demo 4.1/5.0, Speed and Responsiveness 3.9/5.0, and Navigation Ease 3.3/5.0, along with feature-adoption percentages and recommendation percentages. The paper explicitly does not report Text-to-SQL translation accuracy, graph recommendation precision/recall, measured latency, insight extraction quality, or statistical comparison to baseline tools (Fernando et al., 26 Sep 2025). This suggests why later work emphasizes deterministic, reference-based, and image-based objective metrics for visualization generation.

6. Structural and symbolic metrics for SVG and diagram generation

For text-to-SVG generation, SVGauge defines a human-aligned, reference-based metric by combining visual fidelity and semantic consistency. Visual fidelity rasterizes the reference SVG $\mathcal{E}$9 and generated SVG $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$0, extracts SigLIP embeddings, applies PCA/whitening for domain alignment, and computes cosine similarity in the whitened space:

$$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$1

Semantic consistency “captures whether the generated SVG truly reflects the prompt’s meaning” by captioning back the generated image with BLIP-2 and comparing prompt and caption in a combined SBERT+TF-IDF space, yielding

$$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$2

The final score is

$$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$3

with grid search selecting $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$4 and $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$5 for maximal Spearman correlation on the held-out set (Zini et al., 8 Sep 2025).

The evaluation uses the SHE benchmark, containing 333 “gold” SVG images and their auto-generated prompts, with 2,461 valid SVGs produced by eight zero-shot LLM-based SVG generators. A pool of 40 human annotators rated how well generated SVGs match prompts on a 1–5 Likert scale, with each image receiving on average about 8 ratings. At the system level, SVGauge reaches Spearman $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$6 and Kendall’s $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$7, compared with CLIPScore +80.9\% Spearman and +64.2\% Kendall. At the instance level, SVGauge (ref-based) reports Spearman $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$8, versus CLIPScore +37.8\% and SVGauge (ref-free, $$C_{\mathrm{lex}}(T) = \frac{\bigl|\{\text{distinct }n\text{-grams in }T\}\bigr|}{\bigl|\{\text{all }n\text{-grams in }T\}\bigr|},$$9) +36.5\% (Zini et al., 8 Sep 2025).

A second line of work evaluates SVGs structurally through element-level leave-one-out (LOO) analysis. Given rendered elements $$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$0, the framework renders the full SVG and one ablation per element. With a differentiable reference-based similarity score $$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$1, the LOO delta for element $$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$2 is

$$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$3

and the pixel-difference mask is

$$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$4

These signals support zero-shot artifact detection, concept–element attribution, and four structural metrics: purity, coverage, compactness, and locality. The attribution matrix is

$$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$5

with $$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$6, and per-element purity is

$$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$7

At the SVG level, purity averages over active elements; coverage measures the fraction of concepts with at least one assigned element; compactness uses a normalized Herfindahl index over concept attributions; and locality measures how clustered concept-elements are in file z-order (Zhu et al., 9 Apr 2026).

The validation covers 300 SVGs across simple/medium/complex tiers, five generation systems, and roughly 19,000 edits over five edit types. By flagging the 3 most harmful elements with $$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$8, the method achieves F1 ≥ 0.87, at least 0.17 better than baselines, and removing them yields +0.028 SSIM. On the complex tier, reported purities are Claude 0.70, vtracer 0.68, Gemini 0.67, GPT-4o 0.66, Qwen3 0.64, and Source SVG 0.60. Edit precision tracks purity, with Claude 0.80, vtracer 0.80, Gemini 0.79, GPT-4 0.76, Qwen3 0.74, and Source 0.56. Purity correlates with edit precision at the SVG level with $$C_{\mathrm{nov}}(T) = 1 - \max_{R\in\mathcal{D}_{\mathrm{train}}}\cos\!\bigl(f_{\mathrm{text}}(T),\,f_{\mathrm{text}}(R)\bigr).$$9, $V=\{v_1,\dots,v_M\}$00 for color, delete, move, and scale edits, and $V=\{v_1,\dots,v_M\}$01, $V=\{v_1,\dots,v_M\}$02 for regroup after Bonferroni correction (Zhu et al., 9 Apr 2026).

For Image→PlantUML conversion, the VisuGen framework defines component-wise Insertion, Deletion, and Substitution rates over extracted sets of nodes, arrows/edges, messages, and structural constructs such as notes, boxes, groups, and participant declarations. For any class $V=\{v_1,\dots,v_M\}$03,

$V=\{v_1,\dots,v_M\}$04

Specialized edge metrics include a Direction-Change Rate and Type-Change Rate for flipped arrow direction and solid-vs-dashed mismatches. The comparison is implemented through git diff --unified=0, regex pre-filtering, pairwise Levenshtein distances, and Jonker–Volgenant assignment via SciPy’s linear_sum_assignment (Ranjani et al., 15 Sep 2025).

The dataset is drawn from 3GPP Release-18 documents, with about 14,000 images total, of which 32% (≈4,010) are sequence diagrams. The evaluation uses a hand-selected subset of 50 diagrams spanning 1–20 to 51–100 puml lines and totaling about 2,500 ground-truth lines. On these diagrams, Claude 3.7 Sonnet and GPT-4 Vision are compared. Claude reports lower average error rates on basic elements; for example, for Nodes ($V=\{v_1,\dots,v_M\}$05) the insertion/deletion/substitution rates are 13.0 / 15.8 / 12.6, compared with 19.8 / 18.7 / 34.1 for GPT-4. For Messages ($V=\{v_1,\dots,v_M\}$06) they are 13.1 / 14.9 / 11.2 versus 19.7 / 17.9 / 39.3. Both models perform poorly on rare or complex constructs: GPT-4 deletes all boxes, while Claude reports 42.1 / 52.6 / 21.1 for box insertion/deletion/substitution (Ranjani et al., 15 Sep 2025).

Across these SVG and diagram settings, the common principle is structural accountability. Visual similarity alone cannot identify which elements are harmful, which code fragments support meaningful edits, or which symbolic components were inserted, deleted, or renamed incorrectly. The combined evidence from SVGauge, LOO structural metrics, and PlantUML component-wise rates indicates that human-aligned evaluation for visual generation increasingly depends on decomposing outputs into interpretable substructures rather than treating the artifact as an undifferentiated image (Zini et al., 8 Sep 2025, Zhu et al., 9 Apr 2026, Ranjani et al., 15 Sep 2025).