DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation

Abstract: Drama script continuation requires models to maintain character consistency, advance plot coherently, and preserve dramatic structurecapabilities that existing benchmarks fail to evaluate comprehensively. We present DramaBench, the first large-scale benchmark for evaluating drama script continuation across six independent dimensions: Format Standards, Narrative Efficiency, Character Consistency, Emotional Depth, Logic Consistency, and Conflict Handling. Our framework combines rulebased analysis with LLM-based labeling and statistical metrics, ensuring objective and reproducible evaluation. We conduct comprehensive evaluation of 8 state-of-the-art LLMs on 1,103 scripts (8,824 evaluations total), with rigorous statistical significance testing (252 pairwise comparisons, 65.9% significant) and human validation (188 scripts, substantial agreement on 3/5 dimensions). Our ablation studies confirm all six dimensions capture independent quality aspects (mean | r | = 0.020). DramaBench provides actionable, dimensionspecific feedback for model improvement and establishes a rigorous standard for creative writing evaluation.

• The paper introduces a six-dimensional framework assessing format, narrative, character, emotion, logic, and conflict in drama script continuation.
• It utilizes a novel LLM Labeling + Statistical Analysis pipeline to replace subjective scoring with objective, reproducible metrics.
• Experimental results across eight LLMs reveal distinct strengths and weaknesses, highlighting actionable improvement pathways for creative text generation.

DramaBench: Multi-Dimensional Evaluation for Drama Script Continuation

Introduction

DramaBench presents a comprehensive evaluation framework tailored for drama script continuation, targeting the unique challenges of creative writing for screenplays. Unlike existing benchmarks that predominantly focus on aggregate or single-dimensional metrics, DramaBench introduces a six-dimensional structure capable of capturing orthogonal aspects of script quality. The framework is motivated by the recognition that drama continuation demands more than generic language modeling skills: it requires holistic management of character, plot, emotion, logic, conflict, and strict adherence to script formatting. This methodology advances both the rigor and granularity of automated creative writing evaluation.

Figure 1: Overview of the DramaBench evaluation framework, illustrating the three-stage pipeline: contextual task input, six independent evaluation axes, and the structured LLM labeling followed by aggregation into objective metrics for feedback and reproducibility.

Framework Design and Evaluation Dimensions

DramaBench consists of (1) a curated dataset of 1,103 professional English short-form scripts, (2) six orthogonal evaluation dimensions, and (3) a novel LLM Labeling + Statistical Analysis pipeline replacing subjective scoring with robust, structured label extraction. The six evaluation axes are:

1. Format Standards: Rule-based evaluation of adherence to Fountain screenplay format, including error rate, dialogue-action balance, and narrative style ratios;
2. Narrative Efficiency: Event-level extraction of “driver,” “static,” and “redundant” narrative beats; key metrics include Effective Narrative Rate and beats density;
3. Character Consistency: Dialogue-level assessment of persona fidelity (Out-Of-Character rate, voice distinctiveness) using LLM-generated persona profiles;
4. Emotional Depth: Scene-level mapping of emotional arcs using the Valence-Arousal model, detection of emotional shifts and complex layering;
5. Logic Consistency: Fact-level verification against extracted hard constraints from prior context, tracking logical violations and context memory;
6. Conflict Handling: Global assessment of escalation, twist, pause, resolution, or dropping of core conflicts—ensuring alignment with dramatic convention.

The LLM annotator (Qwen3-Max) is leveraged for dimensions 2–6, ensuring categorical outputs that are objective, reproducible, and usable for downstream model training.

Experimental Setup

Eight SOTA LLMs (Claude Opus 4.5, DeepSeek v3.2, GLM-4.6, Gemini 3 Pro Preview, Kimi K2 Thinking, MiniMax M2, GPT-5.2, Qwen3-Max) were evaluated across 8,824 model-script continuations. All models were subjected to identical prompts, ensuring that observed differences are attributable to model capability and not task instruction variance.

Statistical rigor underpinning the results includes 252 pairwise Mann-Whitney U comparisons (with FDR correction, 65.9% significant), human validation on 188 scripts (substantial agreement in 3/5 dimensions), and comprehensive ablation/correlation analysis to confirm independence between evaluation axes.

Results and Comparative Analysis

Model Capability Profiling

Performance profiling reveals distinct specialization patterns across models:

• Format compliance is universal: All models reached 0% Fountain error rate, demonstrating that structure can be fully learned via prompt without architectural modification.
• GPT-5.2 demonstrates overall robustness: Highest ranks in narrative, character, and logic; maintains strong balance but does not dominate in emotion or conflict.
• Qwen3-Max specializes in emotional depth: Achieves unprecedented rates of emotional arc with nuanced complex emotions, but lags in narrative density.
• Gemini 3 Pro excels at conflict handling: Best escalation and multi-threaded conflict scores, but emotional variance is limited and ranks lowest in emotional depth.

  Figure 2: Radar chart visualizing model performances across the six dimensions, highlighting divergent capability profiles and the necessity of multi-axis evaluation.

Error Distribution and Taxonomy

Error taxonomy from 10,850 annotated failures reveals Dialogue-Action Imbalance and Low Information Gain as predominant weak points. There is strong evidence that errors are model-specific: GPT-5.2 demonstrates minimal error rates across categories, while models like GLM-4.6 and Qwen3-Max manifest consistent patterns in logic violations and dialogue imbalance, respectively.

Figure 3: Distribution of the top 10 error types, showing the predominance of Dialogue-Action Imbalance and Low Information Gain across all evaluated models.

Correlation and Ablation Analysis

Spearman correlation analysis confirms dimension independence ($|r|=0.014$ mean absolute correlation), validating that Format, Narrative, Character, Emotion, Logic, and Conflict capture non-redundant quality aspects. Notably, Format is structurally orthogonal to the other axes, while Emotional Depth and Conflict Handling are empirically shown to be uncoupled from foundational capabilities like Narrative Efficiency and Logic Consistency.

Figure 4: Spearman correlation matrix across five content dimensions (excluding Format), confirming minimal cross-axis redundancy and high independence.

Human Agreement and Evaluation Limits

Human validation reveals moderate-to-substantial agreement in logic, emotion, and conflict annotation (Cohen's $\kappa \approx 0.42$–$0.53$), but evaluator-specific bias emerges in narrative and character labeling. This highlights both strengths and necessary cautions: while the structured framework ensures reproducibility, ensemble-based or multi-LLM instantiation will enhance future robustness for subjective axes.

Implications and Future Directions

DramaBench establishes a rigorous paradigm for creative writing evaluation, directly informing several key areas:

• Actionable improvement pathways: Dimension-specific scoring and systematic error classification identify clear avenues for targeted training (e.g., context retention for logic, persona grounding for character, event density for narrative).
• Reproducibility and closed-loop training: Structured labels facilitate both transparency and direct re-use as negative/positive supervision for reinforcement or preference-based fine-tuning.
• Model specialization: SOTA LLMs have developed divergent proficiencies—multi-dimensional evaluators are now required to credibly compare large models, especially as single-score benchmarks obscure key failures.
• Generalizability: The pipeline provides an extensible template for evaluation in other creative or structured generation domains (e.g., novel writing, poetry, dialogue-heavy narratives).

Advancing DramaBench involves scaling human verification, integrating ensemble-based LLM annotation, cross-linguistic extension, and adapting schemas for new genres and longer-form scripts.

Conclusion

DramaBench marks a significant advancement in the automated evaluation of drama script generation. Its six-dimensional, label-driven, and statistically robust approach surfaces nuanced deficiencies and proficiencies in modern LLMs, demonstrating that universal excellence remains elusive. By decoupling structural compliance from creative and logical depth, DramaBench enables actionable, reproducible, and fine-grained assessment, laying the groundwork for future research and development in controllable, creative text generation.

---

Reference:

DramaBench: A Six-Dimensional Evaluation Framework for Drama Script Continuation (2512.19012)