Interactive Creative-Writing Benchmarks

• Interactive creative-writing benchmarks are specialized evaluation frameworks that assess creative text and multimodal generation through interactive, multi-step protocols.
• They employ game-like prompts, constraint-specific grading, and peer review to capture nuances such as originality, emotional resonance, and stylistic divergence.
• These benchmarks integrate automated scoring, LLM-based reasoning, and human annotation to provide actionable insights on creative performance under varied conditions.

Interactive creative-writing benchmarks are evaluation frameworks and datasets specifically designed to assess and guide the creative text (and increasingly, multimodal) generation capabilities of LLMs. Unlike standard generative benchmarks—which typically focus on coherence, grammaticality, or factuality—interactive creative-writing benchmarks directly probe for attributes such as originality, conceptual blending, stylistic nuance, emotional resonance, and creative divergence. These benchmarks operate through interactive protocols (multi-step, peer-involved, or causality-aware procedures) and systematically isolate subjective quality dimensions, addressing the limitations of one-shot or purely objective text evaluations.

1. Evolution and Motivations

The demand for interactive creative-writing benchmarks emerged from recognition that conventional leaderboards, such as those for logic or factual completion (e.g., MMLU, MMMU), do not capture the compositional, associative, and subjective aspects intrinsic to creative generation. Benchmarks like ROCStories and UniEval mainly evaluate general narrative coherence or objective measures, failing to stress uniquely creative skills such as humor, remote association, dramatic twist, or abstract multimodal unification (Huang et al., 25 Jan 2025, Ma et al., 22 Dec 2025). Multi-turn formats, peer-review mechanisms, and domain-specialized test suites now form the core of interactive creative-writing evaluation, targeting processes such as divergent ideation, concept blending, counterfactual reasoning, and creative constraint satisfaction.

2. Platforms, Protocols, and Task Structures

2.1 Game- and Scenario-Led Frameworks

Benchmarks such as LoTbench and the Oogiri Game Platform operationalize creativity via game-like prompts (e.g., Oogiri’s image-to-text punchlines, text riddles), maximizing opportunities for associative leaps (“Leap-of-Thought”) and humorous or surprising responses. The Oogiri-GO dataset, a multilingual corpus of ≈130K samples annotated with human preference (“likes”), supports standard ranking and selection as well as causality-aware evaluation (Huang et al., 25 Jan 2025).

2.2 Script Continuation and Dramatic Structure

DramaBench defines creative script continuation through six orthogonal evaluation dimensions: format standards, narrative efficiency, character consistency, emotional depth, logic consistency, and conflict handling. This partitioning enables targeted assessment of dramatic competence such as plot advancement, persona stability, and scene-anchored emotional arcs, moving beyond generic story or dialogue generation (Ma et al., 22 Dec 2025).

2.3 Constraint-Specificity Grading

CS⁴ (“Comparing the Skill of Creating Stories by Controlling the Synthesized Constraint Specificity”) formulates creativity as model robustness under increasing prompt specificity. By automatically generating up to 39 atomic constraints per prompt, CS⁴ quantifies creativity as the quality retention under hard-to-satisfy, combinatorial requirements. This framework discourages rote training-data regurgitation and highlights generalization under novel constraint sets (Atmakuru et al., 2024).

2.4 Peer Review and Distributed Critique

LLM Review introduces a Blind Peer Review protocol: multiple model agents produce independent drafts, then iteratively review each other’s initial outputs while revising in parallel, without exposure to peers’ revisions. This preserves divergent creative trajectories while incorporating multi-perspective, abstract feedback. The SciFi-100 benchmark exemplifies this approach in science fiction writing, combining automated LLM-as-a-judge, human annotation, and algorithmic novelty scoring (Li et al., 12 Jan 2026).

3. Architecture and Metrics

3.1 Causality-Aware Interactive Scoring

LoTbench departs from one-shot evaluation by engaging models in up to $N$ interactive rounds. At each round, the model attempts a masked-fill in a human-level creative response (HHCR). Causal intervention is performed by swapping key response phrases and checking for “DAESO” (Different Approach, Equally Satisfactory Outcome) via LLM-based reasoning over chain-of-thought graphs. The overall creativity score $S_c$ measures the exponential decay in “rounds to creative insight” (Huang et al., 25 Jan 2025):

$$S_c = \frac{1}{mn}\sum_{j=1}^m\sum_{r=1}^n \exp(-\alpha_c t_r^{(j)}), \ \alpha_c=0.2$$

3.2 Dynamic Criteria and Critic-Model Scoring

WritingBench introduces a query-dependent dynamic criteria framework: for each input, an LLM generates $k$ scoring criteria Cq, each with a name, description, and rubric. A fine-tuned critic model scores outputs per criterion $s_i$ on a 1–10 scale and provides justifications. Aggregated scores are computed as averages or weighted sums (Wu et al., 7 Mar 2025):

$S = \frac{1}{k}\sum_{i=1}^k s_i$

3.3 Constraint Satisfaction and Creativity Degradation

CS⁴ calculates creativity via the product of constraint satisfaction ratio $R_{sat}(n)$ and normalized coherence $C_{norm}(n)$ at increasing constraint levels $n$:

$QUC_n = C_{norm}(n) \times R_{sat}(n)$

A Relative Creativity Score $RCS_{m,n} = QUC_m - QUC_n$ captures how gracefully a model’s output quality degrades under increased specificity (Atmakuru et al., 2024).

3.4 Multi-Dimensional, Modular Scoring

DramaBench implements independent metrics for each of its six evaluation dimensions, for instance, Effective Narrative Rate (ENR), Out-Of-Character Rate (OOC), Arc Score, Logic Break Rate, and Conflict Score—with each metric designed for interpretability and optimization in targeted model submodules (Ma et al., 22 Dec 2025).

3.5 Automatic, LLM, and Human Judgment

Benchmarking protocols combine direct human annotation (e.g., expert pairwise ranking), LLM-as-a-judge rubrics, and rule-based or reference-based novelty scores (e.g., KL divergence, surprisal, semantic embedding distance) (Li et al., 12 Jan 2026, Chen et al., 22 Aug 2025).

4. Subjectivity, Genre, and Multimodal Expansion

4.1 Robustness across Subjective Dimensions

WritingPreferenceBench isolates subjective components by removing all objective confounds (grammar, factuality, length) and measuring model scoring variance across genres. Generative reward models with explicit reasoning achieve up to 81.8% accuracy on human-annotated creative preference pairs, versus ≈53% for standard sequence-classification RLHF models. Extreme genre variance and the insensitivity of large model scale to subjective scores highlight the need for reasoning-augmented and genre-diverse benchmarks (Ying et al., 16 Oct 2025).

4.2 Multimodal Creative Writing

FlexMUSE and ArtMUSE represent a shift to multimodal creative benchmarks, demanding semantic alignment, flexible cross-modal grounding, and creative expression across text and images. Architectures integrate text-to-image (T2I) modules, semantic alignment gates (msaGate), and cross-modality fusion with attention. FlexMUSE evaluation covers both automatic (ROUGE, BERTScore) and subjective (reference-free/aware style, creativity, coherence) scoring, outperforming baselines by large margins on creativity and multimodal richness (Chen et al., 22 Aug 2025).

5. Experimental Findings and Comparative Summaries

Benchmark	Protocol Highlights	Key Distinctions
LoTbench (Huang et al., 25 Jan 2025)	Interactive, DAESO causal chaining	Quantifies “creativity cost,” aligns with human cognition
WritingBench (Wu et al., 7 Mar 2025)	Dynamic criteria, per-dimension critic	Modular, interactive and human-in-the-loop components
CS⁴ (Atmakuru et al., 2024)	Prompt-specificity, constraint grading	Measures creativity under increasing, atomic constraints
WritingPrefBench (Ying et al., 16 Oct 2025)	Subjective pref. pairs, genre isolation	Chain-of-thought boosts subjective preference modeling
DramaBench (Ma et al., 22 Dec 2025)	Six-dimensional, LLM+rule analytics	Fine-grained, drama/script continuation focus
LLM Review / SciFi-100 (Li et al., 12 Jan 2026)	Peer-review multi-agent, novelty metrics	Preserves divergence, boosts novelty via blind critique
FlexMUSE / ArtMUSE (Chen et al., 22 Aug 2025)	Multimodal, semantic alignment/fusion	Flexible cross-modal interaction, creative unification

The comparative evidence demonstrates that interactive, multi-step, or multi-perspective evaluation pipelines both improve measurement of creative potential and supply interpretable signals for model extension.

6. Emerging Design Principles and Theoretical Alignment

Across benchmarks, several consistent methodological principles have been identified:

• Causal or chain-of-thought interventions (LoTbench) better operationalize creativity, echoing abductive and blending theories from psychology (Huang et al., 25 Jan 2025).
• Genre and domain specificity must be systematically controlled, or else models regress to memorized solutions from general corpora (Atmakuru et al., 2024).
• Automated constraint, criterion, or peer-generated feedback loops help disentangle superficial objective compliance from deeper originality and style (Wu et al., 7 Mar 2025, Li et al., 12 Jan 2026).
• Modular, multi-dimensional scoring is necessary for actionable diagnosis and targeted sub-model improvement (e.g., emotional arc, persona modeling, logic, conflict) (Ma et al., 22 Dec 2025).
• Multimodal creative evaluation requires explicit mechanisms for balancing semantic alignment with abstraction, as in FlexMUSE’s stochastic msaGate and multimodal direct preference optimization (Chen et al., 22 Aug 2025).

These benchmarks collectively underpin a transition from static, single-metric leaderboards to robust, interpretable, and human-aligned creative evaluation pipelines for modern generative models.