The Human Creativity Benchmark

Abstract: Modern AI evaluation frameworks treat evaluator disagreement as noise to be resolved. In creative domains, professional disagreement reflects genuine differences in taste, not measurement error. We argue that evaluating creative AI requires preserving two distinct signals: convergence, where professionals align around shared best practices, and divergence, where individual taste legitimately varies. We present the Human Creativity Benchmark (HCB), a benchmark that operationalizes this separation by collecting pairwise preferences, scalar ratings on prompt adherence, usability, and visual appeal, and qualitative rationale from domain professionals. Across 15,000 professional judgments spanning five creative domains and three workflow phases (ideation, mockup, refinement), we find that convergence concentrates on verifiable dimensions like technical correctness and visual hierarchy, while divergence concentrates on taste-driven dimensions like aesthetic direction and conceptual risk. No model excels uniformly across all phases. Collapsing these signals into a single quality metric discards the most actionable information: where models must be correct versus where they should remain steerable.

• The paper introduces a novel framework that distinguishes between objective (prompt adherence) and subjective (visual appeal) axes of creativity.
• It leverages a realistic three-phase workflow (Ideation, Mockup, Refinement) and collects 15,000 expert judgments across multiple creative domains.
• Results reveal phase-specific model performance, suggesting that tailored AI optimization should balance technical consistency with subjective, taste-driven outputs.

The Human Creativity Benchmark: A Multidimensional Evaluation Framework for Generative AI in Professional Creative Work

Introduction and Motivation

Quantifying creativity in AI-generated content has historically been hindered by an operational reliance on objective benchmarks and aggregated quality metrics. These approaches collapse professional subjectivity into noise, neglecting the nuanced interplay of objective standards and individual taste inherent to the creative process. "The Human Creativity Benchmark" (2606.30561) addresses these systemic evaluation deficiencies, proposing a multidimensional, workflow-driven framework that distinguishes between convergent (objective, professional consensus) and divergent (subjective, taste-driven) axes of creative quality.

The benchmark centers on three axes—Prompt Adherence, Usability, and Visual Appeal—each mapped across the Ideation, Mockup, and Refinement stages of real creative workflows. By collecting approximately 15,000 expert judgments over five creative domains (Ad Images, Brand Design, Product Video, Landing Pages, Desktop Apps), the framework interrogates both inter-model differences and intra-domain evaluation dynamics, offering theoretical and practical insights into creative AI performance and deployment.

Methodological Design

The HCB is structured to align with professional design, content, and product development workflows:

• Phases: Ideation captures exploratory, generative divergence; Mockup reflects translation of direction into concrete specification; Refinement targets technical polish and production constraints.
• Domains and Modalities: Evaluated content includes images, video, and code, with state-of-the-art models corresponding to each modality. Model anonymity and randomized ordering control for extrinsic evaluator bias.

Each phase comprises pairwise preference judgments (for subjective rankings), scalar Likert ratings for each axis, and free-text rationales, all within a custom web-based interface tailored to design professional evaluation (Figure 1).

Figure 1: Contra Labs evaluation environment providing domain experts with context-rich, multi-modal judgment tasks.

Evaluator disagreement is intentionally preserved, operationalizing convergence (high inter-rater agreement on verifiable axes such as prompt fidelity) and divergence (legitimate variation in subjective or taste-driven judgments) rather than enforcing consensus via aggregation. This structure is exemplified in Figure 2.

Figure 2: Visualization of divergence (left) and convergence (right) in professional ratings of model outputs for Ad Images Ideation. Note the tight agreement on prompt fidelity versus varied taste in visual appeal.

Quantitative Analysis: Findings Across Domains and Axes

Divergent and Convergent Axes

• Prompt Adherence consistently exhibits the highest inter-rater agreement (convergence), confirming its grounding in technical correctness and fidelity to instructions.
• Visual Appeal displays pronounced divergence, supporting the framework hypothesis that aesthetic judgment and conceptual risk are inherently subject to individual taste (as measured by Kendall’s W and Krippendorff's $\alpha$).
• Usability operates as an intermediary, reflecting a blend of professional standardization and contextual, output-specific judgment.

These trends persist across content modalities, with evaluator comments reinforcing the operational separation: technical failures and missing prompt elements yield consensus, while above-threshold outputs are selected primarily on subjective grounds.

Phase Dynamics in Model Evaluation

Sharp shifts in model preference and evaluator agreement are observed as workflows progress from Ideation to Refinement:

• Early phases (Ideation) see lower agreement, consistent with the exploratory remit and broader latitude for varied creative direction.
• Later phases (Refinement) enforce stricter consensus criteria, prioritizing technical correctness and suitability for production (Figure 3).

  Figure 3: Scalar performance trajectories for video generation, showing model and metric-specific variations across Ideation, Mockup, and Refinement.

No model demonstrates phase-invariant superiority: for instance, Veo3.1 leads in video Ideation but its performance degrades significantly in Refinement where targeted edits and adherence to established concepts become critical. Seedance-v1.5 and Grok-Imagine-Video show inversely mirrored trajectories, reflecting their differing optimization profiles across the creative arc.

Figure 4: Overall pairwise win rates for video models, illustrating phase-specific leadership and absence of a dominant model across the workflow.

Multi-Dimensional Model Comparison

Aggregating model performance into a single "best" obscures workflow- and axis-specific heterogeneity. Figures 11 and 12 (Ad Images) and Figures 23 and 24 (Brand Design) underscore significant cross-phase reordering among model rankings; high correlation between axis-level agreement and pairwise win rates demonstrates the multidimensionality of creative evaluation.

Figure 5: Pairwise win rates for Ad Images, evidencing model-specific strengths and weaknesses by phase.

Figure 6: Ribbon plots showing metric-wise evolution of image model performance through pipeline stages.

Qualitative Evaluation and Professional Insights

Expert free-text rationales reveal strong thematic alignment with the operational axis framework. Failures in prompt adherence (e.g., unreadable or missing elements) are universally penalized, while viability in Usability and Visual Appeal is often debated even among experienced professionals. Notably, evaluators report employing personal aesthetic preference as the primary discriminator once technical minima are met—validating the hypothesis that subjective divergence represents actionable, rather than noisy, information for model design.

Survey data contextualizes the practical reality of AI-assisted creative work: most professionals view AI as a tool for augmentation/co-creation, selectively harnessed across different workflow stages (Figures 6, 7, 8, 9, 10). High trust and adoption rates co-occur with mixed attitudes on AI commoditization of creativity and a general emphasis on AI as assistant rather than replacement.

Figure 7: Distribution of professional sentiment regarding AI’s creative role, reflecting a predominance of mixed and positive views.

Implications and Future Directions

The HCB reframes AI benchmark design for creative domains, advocating retention of both consensus and variability in human judgment to prevent algorithmic homogenization and support professional steerability. For model developers, this enables differentiated optimization: convergent axes can be directly targeted for reliability, while divergent axes should preserve output parametrization and modularity for user control.

This framework also suggests that optimal model selection may be phase- and task-dependent, reinforcing the utility of dynamic, modular creative pipelines over monolithic AI tooling. The findings implicate a need for richer, phase-aware evaluation datasets and upward integration of steerable controls within generative AI systems.

Practical limitations include modest evaluator sample size and workflow simplification relative to real-world, non-linear creative processes. The release of the full dataset on Hugging Face enables further meta-analyses, more granular modeling of creative preference, and potential for longitudinal studies on creative workforce adaptation to evolving generative systems.

Conclusion

The Human Creativity Benchmark establishes a robust, multidimensional paradigm for evaluating AI performance in creative professional contexts. By decomposing creative quality into convergent and divergent axes and structuring evaluation across realistic workflow stages, it advances both the theoretical understanding and the practical assessment of creative AI. This work signals an imperative shift: benchmarking in creative AI must internalize the plurality of professional judgment, facilitating nuanced optimization and model selection, and foregrounding the centrality of human taste and standards in future AI-assisted creative ecosystems.