Automated Creativity Evaluation of Language Models Across Open-Ended Tasks

Abstract: LLMs have achieved remarkable progress in language understanding, reasoning, and generation, sparking growing interest in their creative potential. Realizing this potential requires systematic and scalable methods for evaluating creativity across diverse tasks. However, most existing creativity metrics are tightly coupled to specific tasks, embedding domain assumptions into the evaluation process, and limiting scalability and generality. To address this gap, we introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks. Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment. Divergent creativity is measured using semantic entropy, a reference-free and robust metric for novelty and diversity, validated against human annotations, LLM-based novelty judgments and baseline diversity measures. Convergent creativity is assessed via a novel retrieval-based multi-agent judge framework that delivers context-sensitive evaluation of task fulfilment with over 60% improved efficiency. We validate our framework in three qualitatively distinct domains: problem-solving (MacGyver), research ideation (HypoGen), and creative writing (BookMIA), using a broad suite of LLMs. Empirical results show that our framework reliably captures key facets of creativity, including novelty, diversity, and task fulfilment, and reveal how model properties, such as size, temperature, recency, and reasoning, impact creative performance. Our work establishes a reproducible and generalizable standard for automated LLM creativity evaluation, paving the way for scalable benchmarking and accelerating progress in creative AI.

• The paper introduces a scalable evaluation framework that distinguishes divergent creativity using semantic entropy from convergent creativity assessed by a retrieval-based multi-agent judge.
• It uses semantic entropy to capture meaningful idea diversity in LLM outputs, demonstrating strong correlation with human novelty judgments.
• The retrieval-based multi-agent evaluation reduces token usage by over 60% while maintaining accuracy comparable to individual human assessments.

Automated, Domain-General Evaluation of LLM Creativity Across Open-Ended Tasks

Introduction

The paper "Automated Creativity Evaluation of LLMs Across Open-Ended Tasks" (2606.11762) addresses a fundamental challenge in computational creativity: scalable, generalizable, and automated assessment of the creative capabilities of LLMs on open-ended tasks. Prior approaches are largely domain-restricted, task-specific, or require substantial human annotation, which limits systematic cross-domain benchmarking. This work proposes an evaluation framework that is reference-free, interpretable, and applicable across problem-solving, scientific ideation, and creative writing. It operationalizes creativity via two distinct but complementary facets: divergent creativity—novelty and diversity in idea generation—and convergent creativity—task-oriented solution quality and appropriateness.

Semantic Entropy for Divergent Creativity

The core contribution on the divergent axis is the adaptation of semantic entropy (SE) as a principled, reference-free metric to quantify model exploration in generative reasoning. Unlike naïve diversity measures based on surface-form or token-level variation, SE captures distributional uncertainty over semantically distinct solution types, clustering model outputs by bidirectional entailment.

Models with higher SE allocate probability mass across more meaningfully distinct solution directions, consistent with cognitive flexibility, rather than simply more varied wording.

Figure 1: LLM-generated steps clustered by meaning; semantic entropy distinguishes surface-level rephrasings from genuinely distinct conceptual pathways.

Empirical analyses demonstrate that step-level SE correlates strongly with the number of semantic clusters formed, increases with sampling temperature (controlling for exploratory breadth), and is anticorrelated with embedding-based similarity among candidates. Human annotations corroborate SE's alignment with judgments of idea breadth (Cohen's $\kappa = 0.56$ on MacGyver), outperforming reference diversity metrics like cosine similarity, Self-BLEU, or distinct-n.

Figure 2: Step-level SE tracks semantic branching, increases with sampling temperature, and aligns with human and LLM novelty judgments.

Aggregating SE across steps yields a global creativity estimate congruent with both human and LLM-judge novelty rankings (Spearman $\rho=0.41\text{--}0.47$ across datasets and models). These results affirm SE's utility as a scalable proxy for originality and flexibility, central components in classic creativity theory.

Retrieval-Based Multi-Agent Evaluation for Convergent Creativity

Convergent creativity mandates integrating constraints and expert knowledge to produce functional, context-appropriate, and evaluable solutions. The authors engineer a scalable multi-agent evaluation protocol that overcomes the computational bottlenecks of iterative, full-history discussion frameworks by leveraging a retrieval-based mechanism. Each agent (Problem, Solution, Criterion Analyst) independently embeds its intermediate insights (fragments), which are then selectively retrieved and composed for each subsequent deliberation step, substantially reducing prompt length and token usage (~63% lower than ChatEval).

Figure 3: Retrieval-based multi-agent judge framework integrates distributed LLM perspectives with efficient fragment retrieval for scalable, comprehensive evaluation.

This judge design retains specialization and argumentation diversity, while enabling large-scale, practical deployment. Binary decisions (criterion fulfilled/unfulfilled) are issued for domain-relevant axes—feasibility, safety, effectiveness; scientific accuracy and relevance; coherence, realism, plot completion, depending on the domain.

The framework achieves accuracy on par with individual human annotators (MacGyver: 84.7% vs 81.3–84.7% human; BookMIA: 83.0% vs 75.3–87.0% human) and outperforms single-agent or non-retrieval multi-agent baselines.

Figure 4: Early exit with per-agent confidence enables efficient termination of discussions while maintaining accuracy.

Experimental Setup and Benchmarks

The methodology is validated across three qualitatively distinct open-ended tasks:

• MacGyver: Unconventional physical reasoning, requiring practical, safe, and effective solutions.
• HypoGen: Scientific hypothesis generation/reasoning from ‘bit’ (assumption) to ‘flip’ (novel finding), testing domain-specific logical synthesis.
• BookMIA: Creative writing, evaluating narrative structure, coherence, and originality in stories linking provided start and end sentences.

For each domain, a suite of open-source and proprietary LLMs of varying size and architecture are benchmarked. The evaluation pipeline samples candidate continuations at each step, clusters them for SE, and proceeds with greedy solution construction for convergent assessment.

Figure 5: Unified benchmarking pipeline operating over MacGyver, HypoGen, and BookMIA datasets.

Empirical Findings

Semantic Entropy as a Divergent Creativity Metric

Across tasks and model classes, semantic entropy:

• Correlates with human and LLM-assessed novelty and flexibility.
• Remains robust under varied temperature and sampling regimes.
• Outperforms alternative diversity indices in agreement with human rankings.

Crucially, analysis reveals that increasing model size or recency (e.g., LLaMA 3/3.1/3.3, Vicuna 7B→33B, GPT-3.5→4o) does not systematically yield higher SE. In some cases, SE is stable or modestly declines, suggesting larger or more contemporary models, despite improved correctness, might converge more rapidly to dominant solution modes, implicitly reducing ideational variety.

Figure 6: Convergent creativity, as measured by task fulfilment, scales with model size, recency, and reasoning capability; divergent creativity remains less affected.

Scalable, Reliable Convergent Creativity Assessment

The retrieval-based multi-agent judge:

• Reduces computational cost by over 60% compared to baseline approaches, per token usage statistics.
• Delivers accuracy within human variability, generalizing across domains (notably, where human evaluation is feasible, e.g., creative writing and problem-solving).
• Is robust to hyperparameter choices such as confidence thresholds for early exit.

Empirical Separability of Divergent and Convergent Creativity

Aggregate statistical analysis demonstrates that divergent and convergent axes of creativity are largely uncorrelated across all tasks and models (Spearman |$\rho$|$\leq$0.27 on MacGyver for SE vs. task fulfilment). This finding empirically supports the conceptual distinction drawn in cognitive psychology—the possibility of models (or humans) being capable of flexible ideation yet poor at solution refinement, or vice versa.

Figure 7: No systematic tradeoff or correlation between divergent (SE) and convergent metrics across models on MacGyver.

Theoretical and Practical Implications

This framework decouples creativity assessment from closed-answer or domain-specific canon. SEM-based evaluation captures semantic, not lexical, exploration, providing a more rigorous, reference-free foundation for creativity metrics in LLMs. The retrieval-based judge scales multi-perspective deliberation, establishing a robust protocol for complex solution plausibility assessment.

The empirical independence of divergent and convergent creativity implies that architectural or training interventions—e.g., regularizing toward higher semantic entropy, introducing diversity-encouraging objectives, or modifying sampling at inference—could selectively enhance creative exploration without loss in correctness or utility. Conversely, pure scaling of model parameters and dataset size is, under prevailing training regimes, not sufficient to improve ideational range.

From a benchmarking perspective, the proposed methods provide a reproducible, cross-domain baseline for future LLM creativity research, facilitating meaningful comparison across model families and prompting a richer understanding of the generative reasoning space. This is a critical advancement for both AI systems targeting autonomous scientific discovery and creative applications, as well as for theoretical understanding of how artificial agents instantiate human-like creativity axes.

Limitations and Future Prospects

While the framework is automated, it remains computationally intensive for large-scale experiments due to the N-way sampling and clustering required for SE computation. Reliance on LLM-based judges may introduce biases inherent to pretrained evaluators; highly unconventional solutions may be undervalued, and human interpretability might be limited in specialized or cross-disciplinary scientific domains such as HypoGen.

Prospective directions include:

• Designing model training objectives or reinforcement signals that directly optimize semantic entropy alongside task success.
• Exploring more efficient or scalable semantic clustering and novelty-detection techniques, possibly leveraging contrastive or retrieval-augmented architectures.
• Extending the benchmark suite to new open-ended domains, including code generation, design synthesis, or cross-modal creative tasks.
• Investigating adversarial or meta-level tasks that actively decouple correctness from novelty, challenging models to manifest genuinely unexpected reasoning chains.

Conclusion

This work establishes a scalable, domain-agnostic framework for assessing both divergent and convergent creativity in LLMs, validated across tasks requiring physical reasoning, scientific idea generation, and narrative skill. Semantic entropy operationalizes generative exploration in a reference-free manner, while the retrieval-based multi-agent judge enables efficient and comprehensive task fulfilment evaluation. Their empirical independence confirms the multidimensional nature of creativity, providing the community with both a new methodological toolkit and a foundation for principled further research in computational creativity and generative AI.