GraphicDesignBench (GDB) Benchmark

• GraphicDesignBench (GDB) is a comprehensive benchmark suite designed to evaluate AI models on professional design tasks, covering both low-level visual details and high-level semantic challenges.
• It incorporates real-world design data and design-native metrics to assess multi-dimensional tasks such as layout, typography, infographics, template semantics, and animation.
• GDB exposes current AI limitations in spatial reasoning, typographic fidelity, and structured code generation, guiding research toward improved multimodal design collaboration.

GraphicDesignBench (GDB) is the first comprehensive benchmark suite tailored to the evaluation of AI models on the broad spectrum of professional graphic design tasks. It addresses both low-level visual and high-level semantic challenges specific to design workflows, spanning structured layouts, typographical fidelity, layered compositions, vector graphics manipulation, template synthesis, and temporal reasoning for animation. GDB sets new standards by integrating real-world design data, design-native metrics, and a reproducible framework for both “understanding” and “generation” settings, thereby serving as an essential resource for tracking AI progress and deficiencies in design collaboration contexts (Deganutti et al., 5 Apr 2026).

1. Motivation and Benchmark Scope

Professional graphic design tasks fundamentally differ from natural image generation in their multi-constraint structure, demanding visual, spatial, semantic, and brand-aware reasoning. Existing benchmarks such as FID, CLIPScore, or COCO detection focus on natural-image metrics or generic perceptual tasks, neglecting the multi-faceted and compositional nature of design work, which includes spatial accuracy, fine-grained typography, valid vector code, and animation logic. GDB fills this void by grounding its evaluation in real design artifacts and capturing the “understanding” (perception/parsing) and “generation” (synthesis/editing) spectrum across five axes: layout, typography, infographics, template & design semantics, and animation (Deganutti et al., 5 Apr 2026).

Uniquely, GDB extends far beyond previous LLM-agent benchmarks such as GraphicBench, which concentrates on stepwise planning with language agents for open-ended but narrower design tasks (book covers, business cards, postcards, posters) and measures subjective and objective outcomes via the modular GraphicTown agent pipeline (Ki et al., 15 Apr 2025).

2. Task Axes and Dataset Design

GDB structures its nearly 50 distinct tasks along five primary axes, encompassing both perception and generation capabilities required by design-centric AI systems:

• Layout: Aspect-ratio classification, element counting, component-type classification/detection, z-order prediction, decorative-frame detection, intent-conditioned layout generation, layer completion, and adaptive re-layout under constraints. Requires spatial reasoning in dense, multi-element canvases under occlusion and layering.
• Typography: Font family/weight/size/color/alignment/line-height/letter spacing prediction, curved-text and rotation detection, style-range recovery, as well as synthesis and restoration/editing with style/localization constraints. Requires sub-pixel accuracy and discrimination among >160 font classes.
• Infographics (SVG): Perceptual and semantic Q&A on SVGs, bug fixing, code optimization, style editing, and generation (text/image-to-SVG/Lottie, animation synthesis). Tests code validity, structure preservation, and geometric manipulation in vector space.
• Template Semantics: Fine-grained category/sub-category classification, intent parsing, template retrieval, style completion, and recoloring (both in structured JSON and raster forms). Probes attribute grouping, template taxonomy, and structured style transfer.
• Animation: Keyframe ordering, motion-type/duration/extraction, parameterized animation execution, trajectory synthesis, and video generation from briefs. Requires temporal decomposition and mapping of textual/descriptive cues to time-varying design outputs.

All tasks are grounded in the LICA layered-composition dataset (~1,000 templates), which preserves per-component metadata such as bounding boxes, z-order, font details, SVG/Lottie animation code, and aesthetic/style attributes (Deganutti et al., 5 Apr 2026).

3. Evaluation Framework and Metrics

Recognizing the inadequacy of generic image similarity or caption-matching scores, GDB establishes a taxonomy of design-native metrics spanning spatial, semantic, typographic, and structural dimensions:

Axis	Example Metrics	Evaluation Modality
Spatial Accuracy	mIoU, bbox F1, COCO mAP, MAE/MSE	Layout, infographics
Text Fidelity	OCR Accuracy, Font MAE, ΔE, IoU	Typography, layout
Perceptual Quality	LPIPS, SSIM, FID, DreamSim	Generation, inpainting
Structural Validity	JSON/SVG Validity, Compression Ratio	Infographics, template
Rank/Clustering	Kendall’s τ, ARI, AMI, V-measure	Template/semantic tasks

Additional metrics include human-aligned scores (NIMA, ImageReward), semantic alignment (CLIPScore, PickScore, BERTScore), and holistic judge-based pairwise comparison (M-Judge) on design attributes such as aesthetics, clarity, creativity, and usability.

Evaluation is performed in both understanding (discriminative/predictive labeling) and generation settings (structured synthesis, spatial inpainting, code generation), under image, JSON, and multimodal input/output conditions. Deterministic decoding (temperature=0) and standardized prompt templates ensure reproducibility and fair comparison across models (Deganutti et al., 5 Apr 2026).

4. Model Performance and Key Limitations

State-of-the-art closed-source models (e.g., GPT-5.4, Gemini 3.1 Pro/Flash-Image, Claude Opus, Sora, Veo) are benchmarked extensively. Results reveal that while some aspects of high-level semantic understanding are within reach—such as user intent embedding (BERTScore ≈ 88.5–89.6, CosSim ≈ 91.4–92.7) and template matching (pairwise Acc=96.7%)—acute deficiencies remain when tasks demand compositional structure or pixel-level fidelity:

• Layout tasks: Best element counting MAE reaches 5.81 (on ~15.7 elements/layout) and [email protected] for component detection remains at 6.4%, markedly lower than COCO. Layer ordering (τ=0.567) demonstrates weak correlation with instance detection; even minor z-order errors can bury critical elements such as text.
• Typography: Font family predict top-1 Acc=23.7%; fine detail tasks (color, spacing, style-range) see partial tractability, but ~140/167 font families are essentially undetectable by models.
• Infographics: Text-to-SVG SSIM=0.733 (valid=1.0), but complex nested code often produces invalid output or fails semantic structure. Style transfers misapply geometric transforms.
• Animation: Keyframe ordering exact match at 14–16%, motion type classification consistently <13%, with temporal parameter extraction also bottlenecked at order-of-seconds MAE.

Across axes, pixel metrics (e.g., mIoU, LPIPS) fall short of capturing compositional correctness (e.g., style adherence, element grouping, or brand compliance), and models routinely misplace, overflow, or fail to recover masked/occluded elements. Performance disparities are aggravated as compositional and structural requirements increase (Deganutti et al., 5 Apr 2026).

5. Architectural Approaches and Comparison to Language-Agent Frameworks

GDB-based assessments highlight the limits of vision-only or language-guided image synthesis models when applied to professional design contexts. While planning frameworks such as GraphicTown—evaluating multi-agent LLM workflows with role decomposition and explicit expert-tool action selection (e.g., Photoshop, Illustrator, InDesign proxies) (Ki et al., 15 Apr 2025)—can decompose creative requests into multi-step, tool-centric plans, both structure-aware vision models and LLM-planner agents exhibit analogous failure modes:

• Persistent spatial reasoning errors (e.g., bounding box overflows, incorrect z-order)
• Faulty global dependency tracking (e.g., broken chain of outputs in multi-agent pipelines)
• Attribute hallucination or omission under incomplete prompts or ambiguous constraints
• Invalid action or code retrieval in constrained environments (JavaScript snippets, SVG/Lottie output)

This convergence suggests that future progress requires tight multimodal integration (image, layout, code), task decomposition, and feedback mechanisms to resolve under-specified goals.

6. Research Implications and Directions

GDB reveals that the “last mile” of design automation—ensuring spatial, typographic, and structured code precision under real-world compositional constraints—remains unsolved. Critical research avenues include:

• Spatially Grounded Architectures: Embedding geometric and layout reasoning, beyond bounding box coordinates, into backbone models.
• Typographic-Aware Pretraining: Incorporating specialized datasets and auxiliary tasks (OCR, font/classification) to boost typographic perception and fidelity.
• Structured Output Decoding: Transition from flat image generation to architecture capable of producing valid, nested, and editable code artifacts (SVG, JSON, Lottie).
• Temporal/Animation Reasoning: Seq2seq decomposition for keyframe and motion parameter mapping, with explicit temporal control.
• Human Expert Evaluation Integration: Metrics aligned with professional standards, incorporating direct and relative human preference, especially for style, harmony, and usability.
• Layered Data Supervision: Leveraging GDB’s structural annotation for training and curriculum learning in modular, code-generative, and multimodal models (Deganutti et al., 5 Apr 2026).

A plausible implication is that progress on GDB benchmark tasks will drive the evolution of design-collaborative AI systems capable not only of draft generation but also of fine-tuned, specification-compliant output under routine human supervision. GDB’s open protocol and extensible structure encourage transparent reporting, community benchmarks, and iterative refinement of both algorithms and evaluation standards.

7. Significance for Design Automation and Collaborative AI

By directly measuring the operational gaps in structure, fidelity, and usability, GDB provides a reproducible, extensible testbed for the research community to evaluate and improve all facets of AI-enabled design collaboration. GDB’s design-native approach—layer-wise supervision, compositional metrics, and rigorous multi-modal decoupling—clarifies sub-task boundaries, exposes architectural bottlenecks, and provides a practical path for guiding both incremental improvements and paradigm shifts in creative AI modeling for professional contexts (Deganutti et al., 5 Apr 2026).