---
title: Human Creativity Benchmark for Generative AI
url: https://www.emergentmind.com/topics/human-creativity-benchmark-hcb
type: topic
---

# Human Creativity Benchmark for Generative AI

The Human Creativity Benchmark (HCB) is a benchmark and public dataset for evaluating generative AI systems in professional creative contexts such as ad images, brand visuals, product videos, landing pages, and desktop app UIs. Its defining claim is that creative evaluation should not be collapsed into a single quality score: it should preserve both **convergence**, where professionals align around shared, checkable standards, and **divergence**, where professionals legitimately disagree because of taste. Across 15,000 professional judgments spanning five creative domains and three workflow phases, HCB reports that convergence concentrates on technical correctness and visual hierarchy, whereas divergence concentrates on aesthetic direction and conceptual risk [2606.30561].

## 1. Conceptual basis

HCB was proposed against a common evaluation assumption: that evaluator disagreement is noise to be reduced by averaging, majority vote, or reliability filtering. In creative work, the benchmark argues, disagreement is often the substantive signal rather than a defect of measurement. Different designers, art directors, or video editors may reasonably prefer different concepts, moods, or stylistic directions even when they agree on technical quality. HCB therefore separates two signals that many benchmarks conflate: **convergence**, where professionals largely agree because outputs satisfy shared standards such as readability, layout, or prompt fidelity, and **divergence**, where professional disagreement reflects taste and plurality rather than error [2606.30561].

This framing is tied to real creative workflows rather than abstract creativity prompts. HCB structures creative work into **Ideation**, **Mockup**, and **Refinement**, and treats these phases as qualitatively different evaluation regimes. Ideation is for exploring directions and concepts; Mockup turns a chosen direction into a more concrete artifact; Refinement targets consistency, typography, small fixes, and adherence to constraints. The paper’s “sideways martini” diagram is intended to capture this transition from breadth to narrowing focus [2606.30561].

A related implication is that a single global leaderboard obscures what matters operationally. If one averages across domains, phases, axes, and raters, then high-agreement constraints and taste-driven variation are flattened into the same scalar. HCB therefore treats “what models must be correct on” and “where models should remain steerable” as distinct evaluation targets rather than as components of one undifferentiated notion of quality [2606.30561].

## 2. Benchmark composition and annotation scheme

HCB covers **five creative domains** chosen to map common freelance and agency deliverables: **Ad Images**, **Brand Design / Brand Image Assets**, **Ad Video / Product Video**, **Landing Pages**, and **Desktop Apps**. These are paired with generative modalities and model classes: text-to-image or image-to-image for Brand Design and Ad Images; image-to-video for Product or Ad Video; and text-to-code or code-to-code for Landing Pages and Desktop Apps. Prompts at later stages may include prior outputs, such as images, video frames, or code, to reflect iterative workflows [2606.30561].

The evaluation corpus was built with **28 professional creatives** from **13 countries**, drawn from Contra’s network of independent creatives and assigned to domains aligned with their expertise. The study reports **93 prompts across domains and phases**, **80 evaluation sessions**, and about **15,000 professional judgments**, including **5,940 pairwise comparisons**, **5,940 scalar ratings**, and **3,675 qualitative rationales**. The released dataset on Hugging Face, `contra-labs/HCB`, contains **95 prompts**, **380 model outputs**, **3,174 pairwise comparisons**, **2,116 scalar rating records**, and **2,247 free-text rationales**; the paper attributes these differences to filtering incomplete sessions [2606.30561].

HCB collects three complementary annotation types. First, **pairwise preferences** ask raters to choose which of two anonymized outputs they prefer overall for a given prompt and phase, with all six model pairs compared for each prompt. Second, **scalar ratings** score each output independently on **Prompt Adherence**, **Usability**, and **Visual Appeal**, all on Likert 1–5 scales. Prompt Adherence asks how faithful the output is to the prompt; Usability asks how well the output functions in the context of the prompt and campaign; Visual Appeal asks how visually interesting, cohesive, and polished it is. Third, **qualitative rationales** record free-text explanations, later coded semi-automatically with GPT-4o plus human oversight into themes such as typography, composition, realism, and brand fit [2606.30561].

The three scalar axes are deliberately non-redundant. Prompt Adherence is the least subjective and tends to support convergence; Visual Appeal is the most taste-driven and tends to support divergence; Usability occupies an intermediate position. The benchmark emphasizes that these axes are correlated but not interchangeable, and uses them as probes into where professional agreement is structurally likely and where it is not [2606.30561].

## 3. Operationalizing convergence and divergence

HCB does not introduce a new mathematical theory of creativity; instead it repurposes established agreement and preference models and interprets them through the convergence–divergence lens. Pairwise preferences are aggregated with a **Bradley–Terry** model, in which the probability that model \(i\) beats model \(j\) is

\[
P(i \succ j) = \frac{\exp(\beta_i)}{\exp(\beta_i) + \exp(\beta_j)}.
\]

Fitting this model over all pairwise outcomes yields latent strengths \(\beta_i\), which the paper presents as Elo-like scores or win rates per domain and phase. In this formulation, strong agreement on a winner indicates convergence; roughly balanced or rater-dependent outcomes indicate divergence [2606.30561].

For ranking consistency, HCB uses **Kendall’s \(W\)**, the coefficient of concordance, to measure how consistently evaluators rank outputs. The benchmark interprets \(W \approx 0\) as high divergence and larger \(W\) as greater convergence. For scalar ratings it also reports **Krippendorff’s \(\alpha\)** per model, domain, and phase, treating negative values as less consistent than chance and positive values as more reliable than chance. Statistical testing uses the **Friedman test** within each domain and phase to assess whether scalar ratings differ significantly among models [2606.30561].

Empirically, convergence and divergence are not uniformly distributed across axes or workflow stages. Prompt Adherence shows higher agreement because its criteria are concrete and checkable. Visual Appeal shows lower agreement because judgments of style, mood, and polish are inherently subjective. Usability mixes professional standards with taste and generally falls between the two. The benchmark also ties convergence to concrete design properties such as technical correctness, visual hierarchy, readable typography, and structural layout, while linking divergence to aesthetic direction, conceptual risk, mood, and stylistic flourishes [2606.30561].

Phase structure matters as well. In **Ad Images**, Kendall’s \(W\) rises from **0.345** in Ideation to **0.436** in Mockup and **0.549** in Refinement, indicating stronger convergence as outputs move toward production readiness. In **Landing Pages**, the pattern is different: **0.484** in Ideation, **0.293** in Mockup, and **0.333** in Refinement. The paper interprets this as early convergence around a standout layout followed by renewed taste variation once multiple viable layouts exist [2606.30561].

## 4. Model coverage and evaluation regime

HCB evaluates **13 frontier generative models** distributed across image, video, and code-oriented workflows. Image-domain systems include **gpt-image-1.5**, **gemini-3-pro-image(-preview)**, **seedream-4.5**, and **flux-2-pro / flux-2-max**. Video-domain systems include **veo3.1**, **kling-v3.0-pro**, **seedance-v1.5-pro**, and **grok-imagine-video**. UI and web generation systems include **claude-opus-4.6**, **gemini-3.1-pro-preview**, **gpt-5.3-codex**, and **qwen3.5-397b-a17b**. These models are treated as representative state-of-the-art tools rather than as architecturally analyzed objects [2606.30561].

Generation is deliberately constrained to a realistic workflow setting. For each prompt, all four models in the relevant domain produce outputs for each phase. Parameters such as temperature are standardized where applicable, and Mockup and Refinement prompts may include earlier-stage artifacts. Outputs are collected **once per prompt**, not across many seeds, because the benchmark is intended to reflect a practical “one shot per prompt” usage mode rather than exhaustive sampling [2606.30561].

From these outputs, HCB derives several families of measurements. Pairwise judgments produce Bradley–Terry win rates or Elo-like scores by domain and phase. Scalar ratings yield mean scores for each **Model × Domain × Phase × Axis** combination. Agreement statistics quantify whether raters converge or diverge in those judgments. Qualitative rationales are coded into themes and sentiments, allowing the benchmark to connect numerical agreement patterns to concrete evaluator concerns such as typography, realism, composition, or brand fit [2606.30561].

This structure places HCB in a distinct part of the creativity-benchmark landscape. Earlier visual-advertisement work had already argued that subjective creativity assessment benefits from preserving human rating distributions and disagreement rather than reducing them to a single label [2503.00046]. HCB generalizes that intuition to professional workflows, multimodal outputs, and domain-specific production phases [2606.30561].

## 5. Findings and phase-specific performance patterns

A central empirical result is that **no model dominates all phases**. In **Desktop Apps**, **Claude 4.6** has the highest Ideation win rate at **0.600**, whereas **GPT-5.3** leads Refinement at **0.400**, despite a very low Ideation win rate of **0.086**. In **Ad Images**, **GPT-Image-1.5** is strongest in Ideation and Mockup, while **Seedream 4.5** becomes top in Refinement with win rate **0.367** and stronger appeal and usability. In **Product Video**, **Veo 3.1** leads Ideation at **0.417**, **Kling 3.0** and Veo tie at the top in Mockup, and **Grok Imagine** leads Refinement at **0.361** while Veo falls to **0.083**. **Kling 3.0** is the only video model above 50% in all three phases, at **51%**, **61%**, and **52%** respectively [2606.30561].

These handoffs are not treated as noise. The paper interprets them as evidence that different models specialize in different segments of the creative pipeline: some are better at compelling exploratory directions, others at incremental edits, prompt fidelity, or production polish. This directly supports HCB’s claim that phase-conditioned evaluation is more informative than a single cross-phase leaderboard [2606.30561].

The benchmark also maps where professionals converge and diverge. Convergence clusters around prompt adherence, legibility, visual hierarchy, layout clarity, safe margins, and brand compliance. Divergence clusters around judgments such as whether a look feels “premium” or “cheap,” whether a concept is “too edgy” or “fresh and exciting,” or whether a stylistic direction is compelling rather than generic. An Ad Images Ideation example illustrates this split: the same Flux-2-Pro image was praised as “editorial and stylish, like luxury fashion photography” by one evaluator and criticized as “super heavy… rather looks cheap” by another, with corresponding spread in Usability and Visual Appeal ratings [2606.30561].

The benchmark therefore rejects two common simplifications. First, it rejects the view that disagreement is merely annotation error. Second, it rejects the assumption that creativity evaluation yields a globally superior model independent of workflow stage. A plausible implication is that HCB is better suited to model routing, staged toolchains, and steerable interfaces than to single-number ranking [2606.30561].

## 6. Relation to adjacent benchmarks, limitations, and significance

HCB is part of a broader shift in creativity evaluation, but its emphasis is distinctive. Compared to benchmarks that focus on a single modality or a single creativity construct, HCB anchors evaluation in professional workflows, multiple modalities, and explicit separation of convergent from divergent judgment. Other work has decomposed ad creativity into originality and atypicality while modeling rating distributions and disagreement [2503.00046]; process-aware multimodal benchmarks have evaluated creativity across **idea**, **process**, and **product** with expert rubrics in visual design tasks [2511.13626]; and large meta-benchmarks have argued for a general creativity factor \(c\) across dozens of datasets [2607.01152]. HCB differs by making professional disagreement itself a first-class target of analysis [2606.30561].

The benchmark’s limitations are explicit. The three-phase workflow is a simplification of real creative practice, which is often recursive rather than linear. The evaluator pool comprises only **28 professionals**, so generalization across cultures, industries, and seniority levels remains limited. Models are evaluated using **single-sample outputs** rather than broad seed sweeps. Coverage is restricted to **five domains** and a fixed set of prompt topics. Finally, HCB distinguishes convergence from divergence only at the aggregate level; it does not yet model individual preference structures in detail, though the paper points to hierarchical or perspectivist models as a future direction [2606.30561].

Despite these limits, HCB has become a reference point for professional-creative evaluation because it reframes what benchmark scores should mean. It shifts the question from whether a model is simply “good” to whether it is reliable on shared standards, flexible where taste varies, and competent at the particular stage of work in which it is deployed. In that sense, HCB is both a dataset and a measurement framework for creative AI systems under realistic production conditions [2606.30561].

Source: https://www.emergentmind.com/topics/human-creativity-benchmark-hcb