SGP-GenBench: Benchmarking SVG Generation

• SGP-GenBench is a comprehensive benchmark that evaluates LLMs' ability to generate scalable vector graphics from natural language descriptions.
• It measures object fidelity, scene fidelity, and compositionality using metrics like CLIP, DINO, and human preference scores to assess visual and semantic alignment.
• Reinforcement learning is applied to improve open-source LLM performance, significantly enhancing compositional accuracy and structured reasoning in SVG generation.

SGP-GenBench is a comprehensive benchmarking framework introduced to evaluate the capabilities of LLMs in the domain of symbolic graphics programming—specifically, the synthesis of scalable vector graphics (SVG) code from natural-language descriptions. As presented in "Symbolic Graphics Programming with LLMs" (Chen et al., 5 Sep 2025), SGP-GenBench quantifies object fidelity, scene fidelity, and compositional reasoning, thereby providing a structured lens through which to paper cross-modal alignment and the programmatic generation of visually interpretable outputs.

1. Scope and Structure of SGP-GenBench

SGP-GenBench is formulated to rigorously assess the performance of LLMs at generating SVG code, capturing three critical axes:

• Object Fidelity: Evaluates whether single objects, described in captions, are rendered as accurate SVG representations. The SGP-Object-val set contains 930 curated examples focused on individual objects, scored using metrics such as CLIP-Score, DINO-Score (vision-language alignment), and VQA-Score.
• Scene Fidelity: Uses complex natural images and captions (from COCO-val, a subset of COCO 2017) to probe multiplet-object scenes. Metrics include cross-modal similarity scores (CLIP, DINO, VQA) and a Human Preference Score (HPS) for visual-scene match.
• Compositionality: Inspired by T2I-CompBench, this axis includes:
  • Attribute binding (correct association of colors, shapes, and textures with specified objects),
  • Spatial relationships (correct placements and occlusion order in SVG rendering, both in 2D and 3D),
  • Numeracy (accurate instance counting: e.g., for queries requiring 3–10 objects of certain types, both total and per-type counts are quantified).
  • A panel of judge models (LLMs with task-specific grading prompts) execute sub-task evaluations, with numeracy scores composed of total count accuracy, item recognizability, and count-per-item accuracy (weighted 0.2, 0.2, and 0.6).

These axes together provide a multidimensional, interpretable measure of an LLM’s capacity for visual reasoning, semantic binding, and program synthesis.

2. Quantitative Performance Analysis

Benchmarking on SGP-GenBench yields the following technical findings:

• Proprietary LLMs (Claude, Gemini, etc.) exhibit markedly higher scores, achieving 80–90% on compositional metrics such as color binding and numeracy. Scene and object fidelity also favor proprietary models, indicative of advanced semantic parsing and code generation capabilities.
• Open-source LLMs (e.g., Qwen-2.5-7B baseline) underperform on all axes, with raw compositionality scores as low as 8.8. Issues include invalid SVG code, poor attribute binding, and weak multi-object handling.
• Reinforcement Learning Enhancement: When Qwen-2.5-7B is tuned using RL (as described below), performance improves substantially, reaching compositional scores of 60+ and attaining top VQA scores. This demonstrates that targeted alignment and reward-based optimization can close much of the gap with frontier proprietary systems.

The correlation between SGP-GenBench scores and other coding benchmarks suggests that symbolic graphics programming is intertwined with a model’s general capability for structured reasoning and token-level program synthesis.

3. Reinforcement Learning and Reward Engineering

SGP-GenBench’s RL framework is pivotal for upgrading open-source models:

• Problem Setup: The task is cast as a Markov decision process: for caption $c$, the LLM generates sequential SVG tokens $s_1, \ldots, s_T$, which are subsequently rendered by $R(s)$.
• Reward Formulation:
  • Format-validity gate $r_{fmt}(s)$: A binary condition, requiring outputs to conform to the prescribed "<THINK>...</THINK> <ANSWER>...</ANSWER>" template and render without error via CairoSVG.
  • Perceptual rewards $$[ \lambda_{Text} r_{Text}(s,c) + \lambda_{Image} r_{Image}(s,x) ]$$: Text-image semantic alignment is quantified with SigLIP/CLIP, using cosine similarity between the caption and rendered image; optional image-image rewards use DINO to directly compare with ground-truth images.
  • The total reward is $$r(s, c, x) = r_{fmt}(s) \cdot [\lambda_{Text} r_{Text}(s, c) + \lambda_{Image} r_{Image}(s, x)]$$.
• Policy Optimization: A variant of PPO (GRPO, critic-free) is applied, where token-level advantages are normalized and gradient-clipped, ensuring stable reward propagation to the generation policy.

This pipeline rigorously penalizes invalid SVG output and incentivizes precise semantic and compositional match, directly addressing shortcomings in baseline models.

4. Emergent Training Dynamics

Fine-grained analysis of RL training on SGP-GenBench reveals:

• Increased Program Richness: SVGs become longer, with more graphical elements, improving scene detail and compositional depth.
• Object Decomposition: Models learn to represent complex objects as collections of primitives (e.g., a motorcycle becomes wheels, body, lights, handled separately in SVG), enhancing interpretability and controllability.
• Contextual Detail Injection: Models begin to add scene-appropriate but unprompted visual details (e.g., sprinkles on a cake, background embellishments in a beach scene), improving overall coherence.

An observable trend is the use of comment markers and procedural constructs within SVG code, reflective of improved scene reasoning and alignment with the natural-language intent.

5. Cross-Modal Grounding and Interpretability

SGP-GenBench leverages SVG program generation as a controlled modality for studying cross-modal language–vision grounding:

• Semantic Traceability: Unlike pixel generators, SVG code is well-structured and parametrized, allowing post-hoc inspection and direct attribution between language tokens and visual output.
• Format-Validity Gate as Alignment Scaffold: Ensures that only syntactically valid and fully renderable programs are scored, enforcing tight constraints on both code generation and semantic mapping.
• Potential for Multi-modal Reasoning: The benchmark and RL scheme provide a platform for research into more advanced forms of code–vision–language integration, including extensions to 3D symbolic graphics and planning tasks.

6. Future Directions and Research Implications

Findings suggest several promising paths:

• Model Improvement: RL-based reward alignment can substantially enhance open-source LLM performance, making SGP-GenBench an attractive playground for future algorithmic innovation.
• Benchmark Expansion: Extensions may include non-SVG SGPs, richer compositional structures, or reasoning over dynamic scenes.
• Multi-modal Architecture Design: The work points toward hybrid models that explicitly couple language reasoning, vision encoding, and symbolic graphics synthesis.
• Understanding Emergent Reasoning: SGP-GenBench offers a precise probe for emergent capabilities in LLMs, such as procedural decomposition, attribute tracking, and context inference.

7. Technical Artifacts and Key Formulas

Key technical artifacts include the reward function used in RL tuning:

$$r(s, c, x) = r_{fmt}(s) \cdot [\lambda_{Text} r_{Text}(s, c) + \lambda_{Image} r_{Image}(s, x)]$$

where $r_{fmt}(s)$ enforces format and render requirements; $r_{Text}$ and $r_{Image}$ utilize vision–language encoders for cross-modal semantic similarity.

Diagrams in the original paper—such as Figure 1 and Figure 2—exemplify the structure of the benchmark and improvements in procedural SVG synthesis after RL training.

---

SGP-GenBench establishes an authoritative basis for systematically quantifying and improving the capabilities of LLMs in symbolic graphics programming, offering a multi-faceted, interpretable, and extensible framework for research in structured cross-modal program synthesis (Chen et al., 5 Sep 2025).