Human Creativity Benchmark for Generative AI

• HCB is a benchmark and public dataset that evaluates generative AI in professional creative settings by distinguishing between convergence (shared standards) and divergence (subjective taste).
• It segments creative work into Ideation, Mockup, and Refinement phases, reflecting realistic workflows and enabling stage-specific evaluations.
• The framework employs pairwise comparisons, scalar ratings, and qualitative rationales to assess technical precision and subjective aesthetics across five creative domains.

The Human Creativity Benchmark (HCB) is a benchmark and public dataset for evaluating generative AI systems in professional creative contexts such as ad images, brand visuals, product videos, landing pages, and desktop app UIs. Its defining claim is that creative evaluation should not be collapsed into a single quality score: it should preserve both convergence, where professionals align around shared, checkable standards, and divergence, where professionals legitimately disagree because of taste. Across 15,000 professional judgments spanning five creative domains and three workflow phases, HCB reports that convergence concentrates on technical correctness and visual hierarchy, whereas divergence concentrates on aesthetic direction and conceptual risk (Hopkins et al., 29 Jun 2026).

1. Conceptual basis

HCB was proposed against a common evaluation assumption: that evaluator disagreement is noise to be reduced by averaging, majority vote, or reliability filtering. In creative work, the benchmark argues, disagreement is often the substantive signal rather than a defect of measurement. Different designers, art directors, or video editors may reasonably prefer different concepts, moods, or stylistic directions even when they agree on technical quality. HCB therefore separates two signals that many benchmarks conflate: convergence, where professionals largely agree because outputs satisfy shared standards such as readability, layout, or prompt fidelity, and divergence, where professional disagreement reflects taste and plurality rather than error (Hopkins et al., 29 Jun 2026).

This framing is tied to real creative workflows rather than abstract creativity prompts. HCB structures creative work into Ideation, Mockup, and Refinement, and treats these phases as qualitatively different evaluation regimes. Ideation is for exploring directions and concepts; Mockup turns a chosen direction into a more concrete artifact; Refinement targets consistency, typography, small fixes, and adherence to constraints. The paper’s “sideways martini” diagram is intended to capture this transition from breadth to narrowing focus (Hopkins et al., 29 Jun 2026).

A related implication is that a single global leaderboard obscures what matters operationally. If one averages across domains, phases, axes, and raters, then high-agreement constraints and taste-driven variation are flattened into the same scalar. HCB therefore treats “what models must be correct on” and “where models should remain steerable” as distinct evaluation targets rather than as components of one undifferentiated notion of quality (Hopkins et al., 29 Jun 2026).

2. Benchmark composition and annotation scheme

HCB covers five creative domains chosen to map common freelance and agency deliverables: Ad Images, Brand Design / Brand Image Assets, Ad Video / Product Video, Landing Pages, and Desktop Apps. These are paired with generative modalities and model classes: text-to-image or image-to-image for Brand Design and Ad Images; image-to-video for Product or Ad Video; and text-to-code or code-to-code for Landing Pages and Desktop Apps. Prompts at later stages may include prior outputs, such as images, video frames, or code, to reflect iterative workflows (Hopkins et al., 29 Jun 2026).

The evaluation corpus was built with 28 professional creatives from 13 countries, drawn from Contra’s network of independent creatives and assigned to domains aligned with their expertise. The study reports 93 prompts across domains and phases, 80 evaluation sessions, and about 15,000 professional judgments, including 5,940 pairwise comparisons, 5,940 scalar ratings, and 3,675 qualitative rationales. The released dataset on Hugging Face, contra-labs/HCB, contains 95 prompts, 380 model outputs, 3,174 pairwise comparisons, 2,116 scalar rating records, and 2,247 free-text rationales; the paper attributes these differences to filtering incomplete sessions (Hopkins et al., 29 Jun 2026).

HCB collects three complementary annotation types. First, pairwise preferences ask raters to choose which of two anonymized outputs they prefer overall for a given prompt and phase, with all six model pairs compared for each prompt. Second, scalar ratings score each output independently on Prompt Adherence, Usability, and Visual Appeal, all on Likert 1–5 scales. Prompt Adherence asks how faithful the output is to the prompt; Usability asks how well the output functions in the context of the prompt and campaign; Visual Appeal asks how visually interesting, cohesive, and polished it is. Third, qualitative rationales record free-text explanations, later coded semi-automatically with GPT-4o plus human oversight into themes such as typography, composition, realism, and brand fit (Hopkins et al., 29 Jun 2026).

The three scalar axes are deliberately non-redundant. Prompt Adherence is the least subjective and tends to support convergence; Visual Appeal is the most taste-driven and tends to support divergence; Usability occupies an intermediate position. The benchmark emphasizes that these axes are correlated but not interchangeable, and uses them as probes into where professional agreement is structurally likely and where it is not (Hopkins et al., 29 Jun 2026).

3. Operationalizing convergence and divergence

HCB does not introduce a new mathematical theory of creativity; instead it repurposes established agreement and preference models and interprets them through the convergence–divergence lens. Pairwise preferences are aggregated with a Bradley–Terry model, in which the probability that model $i$ beats model $j$ is

$$P(i \succ j) = \frac{\exp(\beta_i)}{\exp(\beta_i) + \exp(\beta_j)}.$$

Fitting this model over all pairwise outcomes yields latent strengths $\beta_i$, which the paper presents as Elo-like scores or win rates per domain and phase. In this formulation, strong agreement on a winner indicates convergence; roughly balanced or rater-dependent outcomes indicate divergence (Hopkins et al., 29 Jun 2026).

For ranking consistency, HCB uses Kendall’s $W$, the coefficient of concordance, to measure how consistently evaluators rank outputs. The benchmark interprets $W \approx 0$ as high divergence and larger $W$ as greater convergence. For scalar ratings it also reports Krippendorff’s $\alpha$ per model, domain, and phase, treating negative values as less consistent than chance and positive values as more reliable than chance. Statistical testing uses the Friedman test within each domain and phase to assess whether scalar ratings differ significantly among models (Hopkins et al., 29 Jun 2026).

Empirically, convergence and divergence are not uniformly distributed across axes or workflow stages. Prompt Adherence shows higher agreement because its criteria are concrete and checkable. Visual Appeal shows lower agreement because judgments of style, mood, and polish are inherently subjective. Usability mixes professional standards with taste and generally falls between the two. The benchmark also ties convergence to concrete design properties such as technical correctness, visual hierarchy, readable typography, and structural layout, while linking divergence to aesthetic direction, conceptual risk, mood, and stylistic flourishes (Hopkins et al., 29 Jun 2026).

Phase structure matters as well. In Ad Images, Kendall’s $W$ rises from 0.345 in Ideation to 0.436 in Mockup and 0.549 in Refinement, indicating stronger convergence as outputs move toward production readiness. In Landing Pages, the pattern is different: 0.484 in Ideation, 0.293 in Mockup, and 0.333 in Refinement. The paper interprets this as early convergence around a standout layout followed by renewed taste variation once multiple viable layouts exist (Hopkins et al., 29 Jun 2026).

4. Model coverage and evaluation regime

HCB evaluates 13 frontier generative models distributed across image, video, and code-oriented workflows. Image-domain systems include gpt-image-1.5, gemini-3-pro-image(-preview), seedream-4.5, and flux-2-pro / flux-2-max. Video-domain systems include veo3.1, kling-v3.0-pro, seedance-v1.5-pro, and grok-imagine-video. UI and web generation systems include claude-opus-4.6, gemini-3.1-pro-preview, gpt-5.3-codex, and qwen3.5-397b-a17b. These models are treated as representative state-of-the-art tools rather than as architecturally analyzed objects (Hopkins et al., 29 Jun 2026).

Generation is deliberately constrained to a realistic workflow setting. For each prompt, all four models in the relevant domain produce outputs for each phase. Parameters such as temperature are standardized where applicable, and Mockup and Refinement prompts may include earlier-stage artifacts. Outputs are collected once per prompt, not across many seeds, because the benchmark is intended to reflect a practical “one shot per prompt” usage mode rather than exhaustive sampling (Hopkins et al., 29 Jun 2026).

From these outputs, HCB derives several families of measurements. Pairwise judgments produce Bradley–Terry win rates or Elo-like scores by domain and phase. Scalar ratings yield mean scores for each Model × Domain × Phase × Axis combination. Agreement statistics quantify whether raters converge or diverge in those judgments. Qualitative rationales are coded into themes and sentiments, allowing the benchmark to connect numerical agreement patterns to concrete evaluator concerns such as typography, realism, composition, or brand fit (Hopkins et al., 29 Jun 2026).

This structure places HCB in a distinct part of the creativity-benchmark landscape. Earlier visual-advertisement work had already argued that subjective creativity assessment benefits from preserving human rating distributions and disagreement rather than reducing them to a single label (Hou et al., 26 Feb 2025). HCB generalizes that intuition to professional workflows, multimodal outputs, and domain-specific production phases (Hopkins et al., 29 Jun 2026).

5. Findings and phase-specific performance patterns

A central empirical result is that no model dominates all phases. In Desktop Apps, Claude 4.6 has the highest Ideation win rate at 0.600, whereas GPT-5.3 leads Refinement at 0.400, despite a very low Ideation win rate of 0.086. In Ad Images, GPT-Image-1.5 is strongest in Ideation and Mockup, while Seedream 4.5 becomes top in Refinement with win rate 0.367 and stronger appeal and usability. In Product Video, Veo 3.1 leads Ideation at 0.417, Kling 3.0 and Veo tie at the top in Mockup, and Grok Imagine leads Refinement at 0.361 while Veo falls to 0.083. Kling 3.0 is the only video model above 50% in all three phases, at 51%, 61%, and 52% respectively (Hopkins et al., 29 Jun 2026).

These handoffs are not treated as noise. The paper interprets them as evidence that different models specialize in different segments of the creative pipeline: some are better at compelling exploratory directions, others at incremental edits, prompt fidelity, or production polish. This directly supports HCB’s claim that phase-conditioned evaluation is more informative than a single cross-phase leaderboard (Hopkins et al., 29 Jun 2026).

The benchmark also maps where professionals converge and diverge. Convergence clusters around prompt adherence, legibility, visual hierarchy, layout clarity, safe margins, and brand compliance. Divergence clusters around judgments such as whether a look feels “premium” or “cheap,” whether a concept is “too edgy” or “fresh and exciting,” or whether a stylistic direction is compelling rather than generic. An Ad Images Ideation example illustrates this split: the same Flux-2-Pro image was praised as “editorial and stylish, like luxury fashion photography” by one evaluator and criticized as “super heavy… rather looks cheap” by another, with corresponding spread in Usability and Visual Appeal ratings (Hopkins et al., 29 Jun 2026).

The benchmark therefore rejects two common simplifications. First, it rejects the view that disagreement is merely annotation error. Second, it rejects the assumption that creativity evaluation yields a globally superior model independent of workflow stage. A plausible implication is that HCB is better suited to model routing, staged toolchains, and steerable interfaces than to single-number ranking (Hopkins et al., 29 Jun 2026).

6. Relation to adjacent benchmarks, limitations, and significance

HCB is part of a broader shift in creativity evaluation, but its emphasis is distinctive. Compared to benchmarks that focus on a single modality or a single creativity construct, HCB anchors evaluation in professional workflows, multiple modalities, and explicit separation of convergent from divergent judgment. Other work has decomposed ad creativity into originality and atypicality while modeling rating distributions and disagreement (Hou et al., 26 Feb 2025); process-aware multimodal benchmarks have evaluated creativity across idea, process, and product with expert rubrics in visual design tasks (Xue et al., 17 Nov 2025); and large meta-benchmarks have argued for a general creativity factor $c$ across dozens of datasets (Beaty et al., 1 Jul 2026). HCB differs by making professional disagreement itself a first-class target of analysis (Hopkins et al., 29 Jun 2026).

The benchmark’s limitations are explicit. The three-phase workflow is a simplification of real creative practice, which is often recursive rather than linear. The evaluator pool comprises only 28 professionals, so generalization across cultures, industries, and seniority levels remains limited. Models are evaluated using single-sample outputs rather than broad seed sweeps. Coverage is restricted to five domains and a fixed set of prompt topics. Finally, HCB distinguishes convergence from divergence only at the aggregate level; it does not yet model individual preference structures in detail, though the paper points to hierarchical or perspectivist models as a future direction (Hopkins et al., 29 Jun 2026).

Despite these limits, HCB has become a reference point for professional-creative evaluation because it reframes what benchmark scores should mean. It shifts the question from whether a model is simply “good” to whether it is reliable on shared standards, flexible where taste varies, and competent at the particular stage of work in which it is deployed. In that sense, HCB is both a dataset and a measurement framework for creative AI systems under realistic production conditions (Hopkins et al., 29 Jun 2026).