Creativity Benchmark Evaluation

Updated 10 July 2026

Creativity Benchmark is a collection of evaluation frameworks that test AI systems for novel and appropriate outputs under domain-specific constraints, including combinational p-creativity and functional diversity.
They employ methodologies like hidden-target reconstruction, constrained open-ended generation, and multimodal sensemaking with rigorous scoring protocols to ensure human-aligned assessments.
Ongoing debates emphasize that a single creativity score is insufficient, leading to multi-dimensional evaluations that guide future improvements in controllable and diverse creative performance.

Taken together, recent work uses the term creativity benchmark for evaluation frameworks that test whether AI systems can produce novel and appropriate solutions, generate multiple distinct outputs, or make non-obvious connections under domain-specific constraints. The literature no longer treats creativity as a single capability: benchmarks target combinational p-creativity, functional diversity, mathematical construction, real-world creative problem-solving, literary transfer, multimodal sensemaking, design evaluation, and marketing ideation, often with distinct scoring protocols and human-alignment requirements (Guzdial et al., 2018, Zhang et al., 7 Apr 2025, Chen et al., 13 May 2025).

1. Historical emergence and scope

A major early reference point is the Creative Invention Benchmark (CrIB), a 2000-problem benchmark distributed evenly across five diverse domains—Painting, Alien Language, Photobashing, Narrative, and Dessert Recipes—designed to evaluate combinational p-creativity, namely the ability to recombine existing knowledge into artifacts novel to the individual agent (Guzdial et al., 2018). CrIB was presented as, to the authors’ knowledge, the first benchmark to test cross-domain combinational creativity, and it established several design principles that persist in later work: a limited initial knowledge base, domain-specific operators, hidden goals, and continuous feedback rather than direct target exposure (Guzdial et al., 2018).

Later benchmarks broadened both modality and construct. NoveltyBench evaluates whether LLMs can produce multiple functionally distinct responses rather than a single high-quality answer (Zhang et al., 7 Apr 2025). DeepMath-Creative shifts the focus to constructive mathematical ability through proofs and counterexamples (Chen et al., 13 May 2025). CresOWLve evaluates creative problem-solving grounded in real-world knowledge rather than contrived riddles (Ismayilzada et al., 3 Apr 2026). GuessBench moves into multimodal “creativity in the wild” by placing vision-LLMs in the role of guessers in Minecraft “Guess the Build” gameplay (Zhu et al., 1 Jun 2025). Creation-MMBench and CreBench target multimodal creative intelligence and human-aligned creativity assessment for MLLMs, while the Human Creativity Benchmark (HCB) evaluates model outputs in professional creative workflows rather than isolated prompts (Fang et al., 18 Mar 2025, Xue et al., 17 Nov 2025, Hopkins et al., 29 Jun 2026).

The field has also produced meta-benchmarks. AGC-Bench was built from a PRISMA-compliant systematic review that screened 3,101 papers, identified 497 unique creativity benchmarks, and onboarded 78 datasets into a HELM-standardized harness (Beaty et al., 1 Jul 2026). This move from single-task evaluation to benchmark aggregation reflects a central conclusion of the literature: creativity is heterogeneous enough that isolated tests are insufficient for comparative model assessment.

2. Competing definitions of creativity

The benchmarks differ first in what they mean by creativity. In CrIB, the operative construct is combinational p-creativity: novelty arises by combining familiar components, and the novelty need only be new to the agent rather than to humanity at large (Guzdial et al., 2018). In NoveltyBench, the emphasis is not artifact invention but functional diversity: two outputs are distinct only if seeing both provides genuinely new utility to a user (Zhang et al., 7 Apr 2025). In DeepMath-Creative, creativity is tied to constructive mathematical behavior—generating a proof or an explicit counterexample—rather than simply reaching a correct answer (Chen et al., 13 May 2025).

Other frameworks explicitly decompose creativity into dimensions. CreativityPrism organizes evaluation around quality, novelty, and diversity, spanning 9 tasks, 3 domains, and 20 task-specific evaluation metrics (Hou et al., 23 Oct 2025). C²-Eval distinguishes convergent creativity, where tasks admit constrained solutions, from divergent creativity, where tasks are open-ended, and evaluates both through Usefulness, Originality, and Surprise (U-O-S) (He et al., 5 Oct 2025). NoveltyBench’s distinction between distinct $_k$ and utility $_k$ likewise separates mere multiplicity from user-relevant diversity (Zhang et al., 7 Apr 2025).

Some work questions whether creativity should be reduced to a single latent trait at all. AGC-Bench reports that factor analysis across 83 models and 6 creative domains recovers a single creativity factor $c$ explaining 81.5\% of variance, while also showing that models rank differently across domains such as writing and scientific ideation (Beaty et al., 1 Jul 2026). HCB, by contrast, argues that evaluator disagreement in creative domains is not measurement noise but a meaningful signal; it therefore separates convergence, where professionals align around shared best practices, from divergence, where individual taste legitimately varies (Hopkins et al., 29 Jun 2026). This is not merely a psychometric detail: it changes what a creativity benchmark is supposed to preserve.

3. Task families and benchmark construction

The construction of creativity benchmarks follows several recurring patterns: hidden-target reconstruction, constrained open-ended generation, real-world puzzle solving, workflow-based professional judgment, and multimodal sensemaking.

CrIB exemplifies hidden-target reconstruction. In each problem, the agent receives a restricted knowledge base and a domain-specific function such as Paint, AddWord, Stamp, or Submit, plus a Clear function and a scoring function returning feedback in $[0,1]$ (Guzdial et al., 2018). The target artifact is unseen, so the system must infer it indirectly. CS $^4$ uses a different anti-memorization strategy: it increases prompt specificity by controlling the number of synthesized story-writing constraints, ranging from 7 to 39, so that models are less able to retell narratives from training data (Atmakuru et al., 2024).

Constructive benchmarks in technical domains often use expert-authored items and manual grading. DeepMath-Creative contains ~179 problems, about 60\% undergraduate-level and 40\% master’s-level, all original, manually crafted by math experts (Chen et al., 13 May 2025). Each item uses a bidirectional format: if a proposition holds, provide a proof; if not, construct a counterexample (Chen et al., 13 May 2025). CreativeBench, in code generation, distinguishes CreativeBench-Combo for combinatorial creativity from CreativeBench-Explore for exploratory creativity, using automated reverse engineering and self-play with constraint stacking (Wang et al., 12 Mar 2026).

Real-world creative problem-solving benchmarks deliberately move beyond synthetic prompts. CresOWLve is built from 2,061 questions curated from “What? Where? When?” and annotates each puzzle with difficulty, knowledge domains, creative reasoning type, and cultural origin (Ismayilzada et al., 3 Apr 2026). CREATE evaluates associative creativity through 931 curated queries that require models to generate multiple factual paths connecting entities via their parametric knowledge (Wadhwa et al., 10 Mar 2026). LiveIdeaBench evaluates scientific idea generation from 1,180 keywords spanning scientific domains, using minimal context rather than rich retrieval or paper-specific inputs (Ruan et al., 2024).

Multimodal creativity benchmarks extend the same logic to image-grounded or workflow-grounded settings. GuessBench uses 1,500 images and 2,000+ problems from Minecraft “Guess the Build,” with static and dynamic settings and partially revealing hints (Zhu et al., 1 Jun 2025). Creation-MMBench contains 765 test cases spanning 51 fine-grained creative tasks with instance-specific evaluation criteria and role-based prompts (Fang et al., 18 Mar 2025). CreBench operationalizes creativity from idea to process to product, with 12 indicators scored on a five-point rubric (Xue et al., 17 Nov 2025). HCB evaluates outputs across five creative domains and three workflow phases—ideation, mockup, and refinement—using professional judgments rather than crowd preferences (Hopkins et al., 29 Jun 2026).

Benchmark	Task setting	Central construct
CrIB	5 hidden-goal invention domains	Combinational p-creativity
NoveltyBench	Multiple LM outputs per prompt	Functional diversity
DeepMath-Creative	Proof / counterexample construction	Mathematical creativity
GuessBench	Minecraft build guessing	Multimodal sensemaking creativity
AGC-Bench	78 standardized datasets	Artificial general creativity
HCB	Professional workflow evaluation	Convergence and divergence

4. Scoring, calibration, and human judgment

Creativity benchmarks use markedly different scoring schemes because they are trying to isolate different failure modes. CrIB defines a normalized creativity score that subtracts the Uncreative Max baseline: $Score = \frac{NScore_a - NScore_u}{400 - NScore_u}$ so that gains achievable without invention are discounted (Guzdial et al., 2018). The purpose is explicit: improvement above the uncreative ceiling should indicate creative recombination rather than brute-force exploitation of the initial knowledge base (Guzdial et al., 2018).

NoveltyBench formalizes diversity at the set level. Its distinct $_k$ metric counts the number of functionally unique equivalence classes in $k$ sampled generations, while utility $_k$ combines novelty and quality under a user-patience parameter $p$ , with only non-redundant generations contributing value (Zhang et al., 7 Apr 2025). CREATE follows a related set-based logic: its creative utility rewards larger sets of high-quality and diverse associative paths, with path quality determined by specificity and factuality and inter-path diversity determined from embedding distances (Wadhwa et al., 10 Mar 2026). CreativeBench makes the tradeoff explicit in another way by defining creativity as the product of quality and novelty, using executable code to separate creativity from hallucination (Wang et al., 12 Mar 2026).

Because many creative outputs are not fully autogradable, several benchmarks rely on LLM-as-a-judge or hybrid expert pipelines. Creation-MMBench combines unitary scoring for Visual Factuality with pairwise comparison against a baseline, and reports Reward and Win Rate under dual evaluation to reduce position bias (Fang et al., 18 Mar 2025). The literary translation paired-task framework uses expert human annotations and UCP-based automatic scoring, with a creativity score defined by successful Creative Shifts minus unacceptable renderings over the total number of Units of Creative Potential (Zhang et al., 20 Apr 2026). CreBench evaluates alignment to human expert scores by Pearson correlation across idea, process, and product dimensions (Xue et al., 17 Nov 2025).

Judge bias is now a central methodological issue. AGC-Bench applies Judge Response Theory (JRT) to correct for judge severity and leniency and fine-tunes AGC-Judge on 48,299 JRT-corrected ratings (Beaty et al., 1 Jul 2026). By contrast, the marketing-focused Creativity Benchmark reports that three LLM-as-judge setups show weak, inconsistent correlations with human rankings and judge-specific biases, leading the authors to conclude that automated judges cannot substitute for human evaluation (Bhat et al., 5 Sep 2025). HCB makes the related but stronger claim that collapsing professional disagreement into a single quality score discards actionable information about where models must be correct and where they should remain steerable (Hopkins et al., 29 Jun 2026).

5. Empirical regularities

Despite heterogeneity in task design, several empirical patterns recur. First, current models often perform substantially worse on creative requirements than on adjacent reasoning or retrieval tasks. NoveltyBench finds that even state-of-the-art LLMs generate fewer than four functionally distinct responses from 10 samples for most diverse prompts, whereas humans regularly achieve 7–8 (Zhang et al., 7 Apr 2025). DeepMath-Creative reports that, even under lenient scoring, the best-performing model achieves merely 70\% accuracy, primarily on basic undergraduate-level constructive tasks, with sharp decline on harder items and failure to provide substantive strategies for open problems (Chen et al., 13 May 2025). CresOWLve shows a consistent creativity gap: models perform much better on factual questions than on creative ones, with drops on English from -6.08\% to -17.21\% depending on model (Ismayilzada et al., 3 Apr 2026).

Second, strong general ability does not reliably transfer to creativity. The literary translation paired-task framework finds that high source-text comprehension does not yield human-level translational creativity; only one model, Mistral-Large, comes close to the human En–Zh creativity score (0.167 vs. 0.246), and only three model–prompt combinations exceed 0.1 (Zhang et al., 20 Apr 2026). LiveIdeaBench reports that scientific idea generation is poorly predicted by standard metrics of general intelligence and that models with large gaps in general intelligence scores can nevertheless achieve comparable creative performance (Ruan et al., 2024). AGC-Bench similarly concludes that creativity is related to but separable from general knowledge and reasoning (Beaty et al., 1 Jul 2026).

Third, scaling does not induce a uniform creativity gain. NoveltyBench reports that larger models within a family often exhibit less diversity than smaller counterparts (Zhang et al., 7 Apr 2025). CreativeBench describes this as “convergence-by-scaling”: larger models become more correct but less divergent, with scaling helping combinatorial creativity more than exploratory creativity (Wang et al., 12 Mar 2026). C²-Eval likewise reports non-monotonicity with model size and larger gains from reasoning augmentation than from scale alone (He et al., 5 Oct 2025).

Fourth, prompting can help, but usually incompletely. NoveltyBench finds that paraphrasing prompts or telling a model to “be creative” has little impact, whereas in-context regeneration can greatly improve diversity, though this is externally forced rather than intrinsic to the single-prompt distribution (Zhang et al., 7 Apr 2025). AGC-Bench reports that prompting models to “be creative” yields a 1.40 SD increase, much larger than enabling explicit reasoning (0.34 SD) (Beaty et al., 1 Jul 2026). CREATE finds only limited gains from simple creative prompting, with iterative regeneration providing the largest consistent boost in utility and distinctiveness (Wadhwa et al., 10 Mar 2026).

Finally, human superiority persists in domains where taste, interpretive transfer, or real-world ambiguity matter. GuessBench reports that even GPT-4o is incorrect on 34\% of instances and that open and API models differ dramatically in average performance (13.87\% vs. 53.93\%) (Zhu et al., 1 Jun 2025). HCB finds that no model excels uniformly across ideation, mockup, and refinement (Hopkins et al., 29 Jun 2026). The marketing Creativity Benchmark finds tightly clustered performance, with top-bottom spread $_k$ 0, implying a head-to-head win probability of only 0.61, so that no model dominates across brands or prompt types (Bhat et al., 5 Sep 2025).

6. Methodological debates and future directions

A persistent controversy is whether creativity can be summarized by a single score. Some benchmarks explicitly resist that reduction. CreativityPrism reports strong correlations between diversity and quality metrics but much weaker correlation of novelty with either, supporting the claim that strong performance in one creativity task or dimension does not necessarily generalize to others (Hou et al., 23 Oct 2025). C²-Eval formalizes this through separate assessments of Usefulness, Originality, and Surprise across convergent and divergent regimes (He et al., 5 Oct 2025). HCB argues more fundamentally that disagreement itself is informative: convergence concentrates on technical correctness and visual hierarchy, while divergence concentrates on aesthetic direction and conceptual risk (Hopkins et al., 29 Jun 2026).

Another debate concerns contamination, memorization, and the boundary between creativity and retrieval. CrIB addresses this by hiding targets and limiting the knowledge base (Guzdial et al., 2018). CS $_k$ 1 raises prompt specificity to hinder retelling from the training corpus and reports that Learning from Human Feedback helps models select better stories from training data but has limited influence on producing creative stories unseen in the corpora (Atmakuru et al., 2024). LoTbench, built around the Oogiri game, addresses information leakage and interpretability through an interactive, causality-aware framework that scores how quickly a model converges toward high-quality human-level creative responses (Huang et al., 25 Jan 2025). CreativeBench uses executable code and automated test cases to distinguish genuine novelty from non-functional hallucination (Wang et al., 12 Mar 2026).

Future work in the benchmark literature is oriented toward broader coverage, better human alignment, and more controllable creativity. AGC-Bench releases a public leaderboard, AGC-Judge, and human data as open infrastructure for large-scale measurement (Beaty et al., 1 Jul 2026). CreBench and CreMIT seek to train MLLMs that understand human-aligned creativity from idea to process to product (Xue et al., 17 Nov 2025). CREward proposes a type-specific reward model for geometry, material, and texture, extending creativity assessment into image formation pipelines and explainable generation (Han et al., 25 Nov 2025). Across these efforts, a shared implication emerges: evaluation must distinguish where models should converge toward professional standards from where they should preserve pluralism, risk, and steerability. This suggests that the future of creativity benchmarking will be less about a universal scalar and more about structured measurement of multiple creative capacities under explicitly different operational definitions.