DesignIntention Benchmark for Graphic Design

• DesignIntention Benchmark is a prompt-based evaluation framework that transforms vague design intentions into complete, editable graphic designs.
• It employs a layered-generation approach and uses both GPT-4V and human evaluations to assess design quality in layout, typography, and content relevance.
• The benchmark supports diverse task formulations—including text-to-template, add-text-to-background, and text de-rendering—to rigorously test design system performance.

Searching arXiv for the benchmark and closely related papers to ground the article. Searching arXiv for "DesignIntention benchmark COLE Accordion layered graphic design". DesignIntention Benchmark, introduced in COLE under the name “DesignerIntention” and later reused by Accordion, is a prompt-based benchmark for evaluating intent-to-design systems in graphic design. It is designed around the problem of transforming a sparse user intention into a complete graphic design, and, in the layered-generation setting, into an editable composition containing backgrounds, objects, and text. In the literature covered here, it serves as the principal public benchmark for layered graphic design generation from intention prompts, especially for text-to-template generation, adding text to an existing background, and text de-rendering (Jia et al., 2023, Chen et al., 8 Jul 2025).

1. Origin and conceptual scope

DesignIntention originates in COLE, which describes it as a benchmark of designer intention prompts for graphic design generation. Accordion explicitly states that DesignIntention is not introduced there, but comes from COLE and is reused as the main public benchmark. The benchmark is therefore best understood as a standardized prompt set and evaluation protocol for testing whether a system can convert a high-level intention into a high-quality graphic design, and, in the case of layered systems, into a high-quality, editable, layered design (Jia et al., 2023, Chen et al., 8 Jul 2025).

Its motivating problem is the gap between real user input and the requirements of many generative systems. COLE frames the benchmark around “vague” user intentions rather than fully engineered prompts, arguing that real users provide short, under-specified goals. Accordion reinterprets the same benchmark from the perspective of layered design generation, using it to test whether a top-down, reference-driven decomposition pipeline can outperform bottom-up, language-first generation. In that reuse, DesignIntention becomes a stress test for visual harmony, informative text generation, and editability through layer decomposition rather than raster-only synthesis (Jia et al., 2023, Chen et al., 8 Jul 2025).

The benchmark is explicitly graphic-design specific. It is not a natural-image benchmark, a poster-layout-only dataset, or a document-layout dataset. COLE positions it as an end-to-end intent-to-graphic-design benchmark; Accordion positions it as the primary external benchmark for layered graphic design, distinct from training-only datasets such as Design39K (Jia et al., 2023, Chen et al., 8 Jul 2025).

2. Prompt set, categories, and benchmark items

COLE describes the benchmark as containing approximately $\sim 100$ intention prompts for each of five core categories: marketing, covers and headers, posts, events, and advertising. In addition, it includes “an extra set of around $\sim 50$ unique, creative user intentions.” Accordion cites the benchmark as providing 500 detailed prompts across various design domains and uses those 500 prompts for evaluation following COLE’s protocol. The two descriptions are compatible in that COLE presents the category construction in approximate terms, while Accordion refers to the 500-prompt evaluation set it actually uses (Jia et al., 2023, Chen et al., 8 Jul 2025).

Each benchmark item minimally contains a category label and an intention prompt. The intention prompt is a short natural-language description of the desired design, including domain, key textual content, medium, and sometimes stylistic hints. COLE’s supplementary material lists the detailed prompt set. For some baseline experiments, COLE also uses a GPT-4 augmented prompt, but that augmented prompt is not the benchmark itself; it is an auxiliary prompt-engineering device for systems that require more detailed textual conditioning (Jia et al., 2023).

The benchmark is flat rather than fully annotated. It does not provide layered ground truth, layout graphs, typography attributes, or reference images as official labels. In COLE, those structures are generated by the system during inference. Accordion preserves that evaluation regime: DesignIntention is used only for evaluation, while its training data come from Design39K, which contains 39,233 layered designs with descriptions and layer attributes and is used only for training (Chen et al., 8 Jul 2025).

A direct consequence is that DesignIntention is an open-ended benchmark. There is no single correct poster, advertisement, or social-media design for a given intention prompt. This is why both COLE and Accordion rely primarily on evaluator-based scoring and human preference rather than pixel-level or layout-IoU comparison against a canonical target (Jia et al., 2023, Chen et al., 8 Jul 2025).

3. Task formulations

COLE’s primary task is intent-to-graphic-design generation: given an intention prompt, produce a graphic design image. In COLE, the output may be a multi-layer editable design; in raster baselines it is typically a single flat image. Accordion reuses the benchmark for several layered-design task variants aligned with the same intention-oriented setup, most prominently text-to-template, adding text to background, and text de-rendering (Jia et al., 2023, Chen et al., 8 Jul 2025).

Task	Input	Expected output
Text-to-template	Natural-language design intention	Complete layered template
Adding text to background	Background image plus caption or intention	Background with new vector text layer
Text de-rendering	Rasterized design containing stylized or nonsensical text	Clean background plus editable text layer

In the text-to-template formulation used by Accordion, the system first expands the short intention into a detailed descriptive prompt, generates a raster reference image $R$, plans the design from $R$ plus OCR and caption signals, and then decomposes the result into layered output $\{O_1,\dots,O_N,B,T\}$, where $O_i$ are object layers, $B$ is the background layer, and $T$ is a vector text layer. The important benchmark property is not the specific pipeline, but the fact that the prompt is intention-level while the output is expected to be a complete, editable design (Chen et al., 8 Jul 2025).

The “adding text to background” subtask evaluates whether a system can place new paragraph-level text regions onto an existing background while preserving aesthetic harmony. Accordion constructs this comparison by taking COLE’s DesignIntention outputs in SVG, removing all text layers, rasterizing the remaining background, and asking COLE and Accordion to add text to the same background. The “text de-rendering” subtask evaluates whether a system can recover an editable text representation from a stylized raster design and optionally edit or translate that text while preserving layout and style (Chen et al., 8 Jul 2025).

Design variation is also discussed in Accordion, but not as an official DesignIntention metric-bearing task. It is presented qualitatively as a downstream capability enabled by the same decomposition machinery (Chen et al., 8 Jul 2025).

4. Evaluation protocol

Because DesignIntention provides prompts rather than layered ground truth, evaluation is predominantly judge-based. COLE evaluates outputs with GPT-4V(ision) and human pairwise preference. Its GPT-4V protocol scores design and layout, content relevance and effectiveness, typography and color scheme, graphics and images, and innovation and originality. Human studies compare matched pairs of outputs on text fidelity, message conveyance, and visual appeal, with approximately 20 general users and 6 professional designers (Jia et al., 2023).

Accordion adopts COLE’s GPT-4V-based protocol as the main quantitative evaluation. For each of the 500 DesignIntention prompts, a design is generated and GPT-4V rates it on five dimensions: design and layout, content relevance, typography and color, graphics and images, and innovation. Each dimension is scored from 0 to 10, and the paper reports the mean over 500 samples. Accordion also reports an additional LAION aesthetic score and a pairwise GPT-4V preference protocol for the “add text to background” setting (Chen et al., 8 Jul 2025).

Accordion’s reported scores on the 500-prompt evaluation are as follows:

Method	Avg. GPT-4V score	Aesthetic score
COLE	6.0	4.72
Open-COLE	6.3	—
Accordion	6.5	4.98

Accordion further reports metric-level scores of 6.7 for design and layout, 7.4 for content relevance, 6.1 for typography and color, 7.3 for graphics and images, and 5.1 for innovation. In the “add text to background” pairwise GPT-4V comparison, Accordion is preferred in 52% of cases and COLE in 48%. COLE’s earlier benchmark results are reported on a different numerical scale, with values such as 80.82 for Layout, 72.53 for Image, 75.46 for Typography, 87.44 for Content, and 88.72 for Innovation for COLE itself; this suggests that the score tables should be interpreted within their respective protocols rather than as a single unified scale (Jia et al., 2023, Chen et al., 8 Jul 2025).

The absence of canonical targets is both a strength and a limitation. It makes the benchmark open-ended and closer to real design briefs, but it also shifts validity toward evaluator prompts, human studies, and rubric design. COLE explicitly notes that GPT-4V excels at evaluating design and layout, content relevance, graphics and images, and innovation, but falls short in assessing typography and color quality (Jia et al., 2023).

5. Use in COLE and Accordion

In COLE, DesignIntention is the primary benchmark for validating a hierarchical generation framework that transforms an intention prompt into a multi-layered and editable design. COLE uses the benchmark to compare against DALL·E 3, SDXL, DeepFloyd/IF, and CanvaGPT. Its reported findings emphasize strong layout and typography performance, improved text fidelity and message conveyance over DALL·E 3 in user studies, and better preference among designers than CanvaGPT (Jia et al., 2023).

Accordion uses the same benchmark to argue for a top-down approach to layered design generation. Instead of building a design bottom-up from language alone, it starts from a visually harmonious raster reference and decomposes that reference into editable layers. On DesignIntention, this setup is reported to produce gains in design and layout, content relevance, typography and color, and graphics and images relative to COLE and Open-COLE, while innovation remains comparable. Accordion also reports that replacing nonsensical text in raster references with prompt-relevant text improves content relevance from 7.0 for Flux-only raster outputs to 7.4 for Accordion layered outputs (Chen et al., 8 Jul 2025).

The benchmark also exposes differences in what systems optimize. COLE’s bottom-up pipeline is organized around hierarchical planning and generation of background, objects, and text. Accordion argues that this can lead to misalignment because the background may be generated before enough space is reserved for text and objects. Its top-down reuse of DesignIntention is intended to test whether global visual context improves spatial alignment, typography, and text density. In the “adding text to background” comparison, COLE tends to produce shorter text, with an average text length of 42.3 characters, while Accordion produces longer text, with an average of 61.7 characters, or about $1.5\times$ more, while still being slightly favored in pairwise GPT-4V judgments (Chen et al., 8 Jul 2025).

Designer studies reinforce the editability dimension that motivates layered design benchmarks. Accordion reports that, in user studies with 29 designers, its text-to-template editability is judged superior to COLE in 73.5% of cases. This does not redefine DesignIntention’s official metrics, but it highlights an important practical reading of the benchmark: success is not only visual plausibility, but also whether the generated design can be revised in a designer-facing workflow (Chen et al., 8 Jul 2025).

6. Position in the benchmark ecosystem and methodological significance

DesignIntention occupies a specific niche within design-oriented evaluation. COLE and Accordion contrast it with poster-layout and document-layout datasets that assume many elements are already given and focus on layout prediction or text-box placement. By contrast, DesignIntention evaluates the entire chain from raw intention text to complete design. That makes it more specific than generic visual-design benchmarks and more end-to-end than layout-only benchmarks (Jia et al., 2023, Chen et al., 8 Jul 2025).

Related work has broadened the idea of a design-intention benchmark in several directions. DEsignBench targets visual design scenarios for text-to-image models; IDEA-Bench evaluates 100 real-world professional design tasks with multimodal inputs and hierarchical requirement-first scoring; UI-Bench measures expert pairwise preference for AI text-to-app tools using a client-delivery question; GraphicDesignBench evaluates layout, typography, infographics, template semantics, and animation using layered templates; DesignQA evaluates engineering requirement understanding across rules documents, CAD, drawings, and compliance reasoning; and TPS-CalcBench operationalizes design intention in safety-critical engineering by separating answer correctness from engineering process quality and explicitly detecting “Right Answer, Wrong Reasoning” (Lin et al., 2023, Liang et al., 2024, Jung et al., 28 Aug 2025, Deganutti et al., 5 Apr 2026, Doris et al., 2024, Zheng et al., 20 Apr 2026).

Within that broader landscape, the named DesignIntention benchmark remains a graphic-design prompt benchmark centered on intent-to-design generation. Its main methodological limitation is that it lacks ground-truth layered outputs, so evaluation relies heavily on LLM/VLM judges and human preference. A plausible implication is that future versions could benefit from clearer benchmark governance of the kind advocated in benchmark-methodology work such as How2Bench, which emphasizes data quality assurance, deduplication, reproducibility materials, and explicit documentation of scope and evaluation procedure (Cao et al., 18 Jan 2025).

Its continuing significance lies in the combination of three properties: it uses intention-level prompts rather than fully specified layouts, it evaluates complete designs rather than isolated components, and, in later work, it serves as the main public testbed for editable layered generation rather than raster-only synthesis. In that sense, DesignIntention marks a transition from generic image-generation evaluation toward benchmarks that measure whether generative systems can satisfy communicative, typographic, spatial, and editing-oriented requirements in a realistic graphic-design workflow (Jia et al., 2023, Chen et al., 8 Jul 2025).