AGC-Bench: Measuring Artificial General Creativity

Abstract: Creativity research has debated whether creativity is domain-specific (e.g., visual, writing, science), and if it is psychometrically separable from general intelligence. Both questions now apply to LLMs, but a unified benchmark of AI creativity remains elusive. We introduce AGC-Bench, an artificial general creativity benchmark built from a systematic review of the AI creativity literature (3,101 papers screened, 497 benchmarks identified), paired with an agentic harness that converts idiosyncratic codebases into HELM-standardized benchmarks. The first release covers 78 datasets spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory -- a psychometric calibration of judge leniency/severity; we then fine-tune Qwen3-30B on the bias-corrected ratings of three frontier LLMs to produce AGC-Judge, an open-weight model that robustly scores new creativity benchmarks it was not trained on. Results reveal frontier models at the top of the AGC-Bench leaderboard, with open models close behind. LLMs show different creative strengths, ranking higher on some domains (e.g., writing) than others (e.g., scientific ideation). Extensive experiments yield three main findings. First, applying factor analysis across 83 LLMs, we recover a single creativity factor 'c', analogous to the 'g' factor of general intelligence, that explains 81.5% of variance, related to but separable from general knowledge/reasoning. Second, we show that prompting models to "be creative" boosts their performance far more than enabling reasoning, evidence that the benchmark tracks creativity over general ability. Third, on a human-matched subset, we find the top human still leads the top LLM on creativity. We release AGC-Bench with a public leaderboard, AGC-Judge, and human data as open infrastructure for measuring AI creativity at scale.

• The paper introduces a unified evaluation framework for artificial general creativity by consolidating 78 diverse benchmarks into a standardized, psychometric approach.
• It employs Judge Response Theory with Bayesian calibration and an AGC-Judge scorer to ensure robust, cost-efficient scoring across multiple creative domains.
• It demonstrates that LLMs exhibit a distinct, domain-general creativity factor, separating creative prowess from conventional intelligence and scale.

Measuring Artificial General Creativity: An Analysis of AGC-Bench

Overview and Motivation

"AGC-Bench: Measuring Artificial General Creativity" (2607.01152) presents the first large-scale, psychometrically-grounded meta-benchmark for artificial general creativity (AGC) in LLMs. The motivation is rooted in the need to systematically assess not whether LLMs are "creative" in isolation, but whether their creative capabilities exhibit domain generality analogous to the psychometric $g$ factor in human intelligence research. Historically, creativity evaluation in AI has been fragmented across hundreds of narrowly-focused benchmarks with inconsistent metrics, precluding robust inferences on the domain-generality or specificity of LLM creative ability. AGC-Bench addresses this by consolidating 78 benchmarks (67 text-only, 11 multimodal), rigorously curated via a PRISMA-compliant systematic literature review, into a unified evaluation and scoring framework.

Benchmark Construction and Methodology

The authors conduct an exhaustive benchmark aggregation pipeline: 3,101 candidate papers are systematically reviewed, yielding 497 unique creativity-relevant benchmarks after a combination of LLM-automated and human-in-the-loop screening. Selected benchmarks undergo agentic onboarding into a standardized, HELM-style evaluation harness. The first release covers 67 text-only datasets spanning six domains—brainstorming, problem solving, STEM, narrative, figurative language, and humor—plus 11 multimodal scenarios. Domain assignments utilize a consensus of LLM annotators, establishing alignment with established human creativity taxonomies. Strict adherence to source-paper prompts and scoring metrics is maintained.

To score open-ended creative tasks, the authors address scaling and reliability limitations of LLM-as-judge protocols by leveraging Judge Response Theory (JRT), a Bayesian graded response model. This accounts for judge severity/discrimination and calibrates scores across a three-LLM judge panel (Gemini-3-Flash, Grok-4.1-Fast, GPT-4.1-mini) under a planned-missing 2-of-3 cell design. The result is robust, rater-invariant model evaluations that preserve leaderboard rankings while eliminating judge- and scale-specific artifacts.

Figure 1: Per-judge severity and composite density before and after JRT calibration; normalization yields a stable leaderboard basis.

An additional artifact is AGC-Judge: a LoRA-fine-tuned Qwen3-30B-based open-weight scorer trained on 48,299 JRT-calibrated ratings, producing scores that match the ensemble of three frontier LLM judges at a fraction of cost and latency.

Figure 2: JRT calibration maintains leaderboard ranks (strong diagonal alignment) and AGC-Judge generalizes robustly to held-out creativity benchmarks.

Empirical Analysis and Main Findings

Evaluation covers 83 LLMs spanning proprietary (Anthropic, OpenAI, Google, Z-AI, Moonshot) and open-weight models, all profiled across the 67 text-only benchmarks. Three primary findings are established:

1. Existence of a General Creativity Factor ($C$):

Factor analysis on a per-(model, domain) performance matrix reveals a strong unidimensional structure: a single component, $C$, explains 81.5% of variance across six creative domains, with high internal consistency ($\alpha = 0.96$), and all domains loading strongly ($[.87, .94]$). This $C$ factor persists after controlling for both fluid reasoning and knowledge, across different scoring protocols, and under bootstrapping of model subsets, indicating stability and nontriviality.

2. Distinctness from Intelligence and Model Scale:

Rank correlations show that $C$ is substantially but not entirely collinear with proxies for fluid intelligence (counterfactual analogies, $r \sim .55$) and crystallized knowledge (MMLU-Pro, $r \sim .62$). Model scale correlates with $C$ among open-weight models ($C$0 for parameter count), but mixture-of-experts designs display diminished returns when normalized by active parameter utilization, implying that size alone is incomplete for predicting creative competence.

Figure 3: Relationship between AGC-Bench composite and total parameter count for open-weight models, revealing only partial alignment.

3. Domain-General versus Domain-Specific Creativity:

Although frontier models (Anthropic/Claude-Opus-4.7, OpenAI/GPT-5.4, Z-AI/GLM-5.1) lead the overall leaderboard, the per-domain rankings expose pronounced specialization: models optimized for one domain (e.g., kimi-k2.5 in problem solving, Claude-Opus-4.6-Fast in figurative language) may not excel in others.

Figure 4: Left—overall top-15 models on AGC-Bench; right—domain-wise performance exposes creative specialization at the top.

Prompting Manipulation Experiments reinforce the validity of the composite: creativity-targeted prompting ("be creative") increases creative scores significantly more than reasoning-targeted toggling, demonstrating that AGC-Bench rewards genuinely creative output rather than general problem-solving.

Human–AI Comparison and External Validity

Using a five-task paired evaluation with 201 humans and 80 LLMs, AGC-Bench reveals that while LLMs are more domain-general in their creative abilities (human $C$1; LLM $C$2, $C$3), the most creative individual human still outperforms the strongest LLM by $C$4 in composite creativity score.

Figure 5: Comparative creativity scores of top-15 humans and LLMs in the paired-task cohort; human creative upper-bound persists.

Furthermore, the AGC-Bench composite is predictive of agreement with independent human creativity ratings from the MuCE dataset, outperforming alternative intelligence and knowledge proxies.

Scoring Infrastructure and Reproducibility

AGC-Bench releases an open-source, reproducible infrastructure covering benchmark definitions, scoring code, domain classification metadata, per-model outputs, and public leaderboards for ongoing evaluation. The pipeline supports modular extension to new creative benchmarks and additional modalities. The AGC-Judge open-weight scorer enables cost- and compute-efficient future benchmarking against the JRT-calibrated peer ensemble.

Implications, Limitations, and Future Directions

Theoretical Implications

The robust empirical $C$5 factor structurally parallels the $C$6 factor in intelligence, providing evidence that current LLMs display strongly domain-general creative capacities that—while related to fluid reasoning and model scale—are separable from both. This supports hypotheses of hierarchical cognitive architecture in scaling LLMs, where emergent properties such as creativity and reasoning, though correlated, can be dissociated through targeted intervention and evaluation design.

Practical Implications

AGC-Bench sets a new empirical gold standard for evaluating AI creativity at both breadth and psychometric rigor. The calibrated judge ensemble and open-weight AGC-Judge scorer can now serve as a foundation for model selection, ablation studies, human-in-the-loop comparison, or benchmarking protocol design for future AI systems focused on creative output.

Limitations

The composite score is inherently cohort-relative without an absolute theoretical ceiling, and only English-centric, text and vision inputs are covered in the current release. Moreover, the benchmark incompletely addresses transformational creativity—a domain that remains methodologically elusive. There is also residual bias in LLM-judge scoring toward LLM-style outputs, which, while attenuated by contemporary fairness interventions, has not been entirely eliminated.

Prospects for Future Work

Key future directions include designing cost-efficient short-form creativity diagnostics, expanding AGC-Bench into underexplored modalities (audio, video), developing metrics for transformational and combinatorial creativity, and leveraging the framework to probe the causal mechanisms underlying creativity emergence in increasingly large-scale and agentic AI systems. There is also substantial scope for multi-lingual and cross-cultural expansion, as well as standardized protocols to further close the gap with human creative upper bounds.

Conclusion

AGC-Bench establishes a psychometric, reproducible, and extensible foundation for measuring artificial general creativity in LLMs at scale. Through rigorous benchmark curation, calibrated scoring, and methodologically robust analysis, it provides compelling evidence for the existence of a general creativity factor in LLMs, while preserving meaningful dissociations from intelligence and model scale. The publicly released infrastructure and ongoing onboarding support its use as a community resource for advancing the scientific study of machine creativity.