SArena Benchmark for SVG Tasks

• SArena Benchmark is a comprehensive evaluation suite for Structured Vector Graphics, standardizing tasks across icons, illustrations, chemistry, and animations.
• It employs systematic task categorizations and multimodal metrics like accuracy, PSNR, FID, and DINO to rigorously evaluate SVG understanding, editing, and generation.
• Baseline results, including a notable +8% accuracy gain in SVG understanding, demonstrate SArena’s impact in advancing SVG-centric multimodal model research.

SArena Benchmark

SArena is a comprehensive, evaluation-only benchmark suite for Structured Vector Graphics (SVG) tasks, specifically designed to support systematic assessment of multimodal LLMs (MLLMs) on SVG understanding, editing, and generation. Developed in conjunction with the SAgoge dataset, SArena standardizes task formulations and evaluation metrics across four SVG domains—icons, long-sequence illustrations, scientific diagrams, and vector animations—spanning a diversity of structural complexity and semantic depth. The benchmark is constructed to address the fragmented landscape of SVG evaluation, supporting granular stratifications by task family, domain, and difficulty, and has been adopted as the core evaluation environment for the InternSVG model family (Wang et al., 13 Oct 2025).

1. Scope and Task Taxonomy

SArena comprises three major task families, each represented across four domains (Icons, Illustration, Chemistry, Animation), with domain-specific sub-benchmarks. Its aim is to cover the spectrum from low-level syntactic manipulations to complex semantic and hierarchical SVG construction.

• SVG Understanding
  • Multiple-Choice QA: Given raw SVG source code (XML), the task is to select the correct answer from four choices, evaluating code-level semantic and structural comprehension.
  • SVG Description: Given a rendered raster image of an SVG, the model outputs a free-form textual description of salient semantic and geometric features.
• SVG Editing (Icons only)
  • Low-level Editing: Input is SVG code plus an instruction (e.g., “rotate 45°”, “change fill #00abff → #D8BFD8”); output is SVG code implementing the instruction, with minimal diff from the original.
  • High-level Editing: Tasks involve semantic color manipulation, style transfer, and more abstract transformations.
• SVG Generation
  • Text-to-SVG: Textual prompt/caption to SVG code.
  • Image-to-SVG: Reference raster image (optionally with instruction) to SVG code.
  • Text-to-SANI: Textual description of an animation to SVG embedding SMIL-compliant animation tags.
  • Video-to-SANI: Source video (sampled frames) plus instruction to SVG animation code.

The benchmark includes approximately 31,000 evaluation cases evenly split across tasks and domains.

2. Dataset Structure and Statistics

SArena is an evaluation-only testbed derived from the SAgoge corpus. All samples are curated through a combination of automated preprocessing and manual review to remove corrupted or ambiguous SVGs. There are no train or validation splits within SArena itself, and each task operates over a fixed set of samples.

Table: SArena Data Statistics (Selected Columns)

Domain	Task	Samples	Average Tokens
Icon	MCQA	6,012	1,197
	Editing	2,000	3,175
	Text-to-SVG	6,013	1,118
	Image-to-SVG	6,013	1,169
Illustration	Text-to-SVG	2,001	8,181
	Image-to-SVG	2,001	8,291
Chemistry	Text-to-SVG	3,003	1,010
	Image-to-SVG	3,003	1,065
Animation	Text-to-SANI	504	1,677
	Video-to-SANI	504	1,719

Average token count directly reflects SVG hierarchical and geometric complexity, particularly for illustration and animation tasks. The SVG vocabulary covers 55 tag tokens, 42 attribute tokens, and fine-grained numeric tokens over a broad range.

3. Task Difficulty Stratification

SArena stratifies tasks by complexity within and across domains:

• Understanding
  • MCQA emphasizes semantic reasoning over raw code (high difficulty).
  • Description tasks demand semantic and geometric summarization from images (medium difficulty).
• Editing (Icons)
  • Simple syntactic edits (translations, rotations) vs. hard semantic and style-transfer operations.
• Generation
  • Icon generation involves short, low-complexity SVG sequences.
  • Illustrations emphasize deep hierarchical and geometric structure.
  • Chemistry benchmarks test domain-specific semantics in molecular diagrams.
  • Animation tasks require temporal dynamics, motion encoding, and SMIL compliance.

Sequence length and attribute diversity scale with task and domain complexity, particularly for illustration and animation benchmarks.

4. Evaluation Protocols and Metrics

All model outputs (SVG/static, SVG/animated) are rendered to images or videos for metric computation. Invalid or unrenderable outputs are replaced by all-black frames to penalize decoding errors.

• Understanding (MCQA)
  • Accuracy: $$\text{Acc} = \frac{\#\text{correct}}{\#\text{total}} \times 100\%$$
• Editing & Image-to-SVG
  • DINO score: Cosine similarity in self-supervised vision model feature space.
  • SSIM: $$\mathrm{SSIM}(x, y) = \frac{(2 \mu_x \mu_y + C_1)(2 \sigma_{xy} + C_2)}{(\mu_x^2 + \mu_y^2 + C_1)(\sigma_x^2 + \sigma_y^2 + C_2)}$$
  • LPIPS: Learned Perceptual Image Patch Similarity.
  • PSNR: $$\mathrm{PSNR} = 10 \log_{10} \Bigl( \frac{I_{\max}^2}{\mathrm{MSE}} \Bigr),\quad \mathrm{MSE} = \frac{1}{N} \sum_{i} (x_i - y_i)^2$$
• Text-to-SVG
  • Frechet Inception Distance (FID): $$\mathrm{FID} = \|\mu_r-\mu_g\|^2 + \mathrm{Tr}\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right)$$
  • FID-CLIP: FID in CLIP feature space.
  • CLIP-T2I / CLIP-I2I: Cosine similarity between text/image embeddings.
• Animation (Text-to-SANI)
  • Fréchet Video Distance (FVD).
  • CLIP-T2V / CLIP-V2V: Cosine similarity in video feature space.

This protocol guarantees consistent, multi-faceted model assessment across syntax, semantics, perceptual similarity, and structure.

5. Baseline Performance Results

InternSVG (8B) achieves the strongest performance among both open-source and proprietary baselines on all main SArena sub-benchmarks. For the Icon domain:

Model	Understanding (Acc.)	Editing (PSNR)	Text-to-SVG (FID)	Image-to-SVG (DINO)
Claude-4-Sonnet	77.1%	–	–	–
InternSVG 8B	85.1%	77.33	8.715	0.949

Notable findings include a +8 point accuracy gain in understanding, a +20 PSNR improvement in editing, an FID reduction in text-to-SVG, and a DINO score increase for image-to-SVG. Similar performance deltas (often 2–15×) are reported for Illustration, Chemistry, and Animation tasks. These metrics are computed over fixed evaluation sets as described in Section 2 (Wang et al., 13 Oct 2025).

6. Representative Task Examples

Representative cases exemplify benchmark scope:

• MCQA Example
  • SVG: <svg ...><circle cx="64" cy="64" r="50" fill="red"/></svg>
  • Question: How many circles are drawn? (Correct: 1)
• Low-level Editing Example
  • Input: red rectangle SVG + instruction “Change fill from red to #0000FF.”
  • Output: <rect ... fill="#0000FF"/>
• Text-to-SVG (Icon) Example
  • Prompt: “A green triangle pointing up with blue outline.”
  • Output: <polygon points="..." fill="green" stroke="blue" .../>
• Animation Example (Text-to-SANI)
  • Prompt: “Animate a circle moving left to right in 2 seconds.”
  • Output: <animateMotion dur="2s" ...><mpath xlink:href="#circlePath"/></animateMotion>

Full visual comparisons are included in (Wang et al., 13 Oct 2025), often highlighting decoding failures or semantic errors in competitor outputs.

7. Context, Limitations, and Significance

SArena unifies SVG evaluation by combining diverse task families, scaling structural complexity, and employing multimodal metrics tailored to each task. Its domain and difficulty stratification enable fine-grained capability analysis, while strict input-output standards support reproducible model comparison. A plausible implication is that positive transfer observed when training models on the entire SAgoge/SArena spectrum can generalize SVG semantic, syntactic, and perceptual skills, as observed in the performance of InternSVG.

By providing a single evaluation environment aligning with the diversity and difficulty of real-world SVG challenges, SArena constitutes a comprehensive and rigorous foundation for SVG-centric MLLM research (Wang et al., 13 Oct 2025).