ConStory-Bench: Evaluating Narrative Consistency

• The paper introduces ConStory-Bench, a novel benchmark that evaluates narrative consistency in ultra-long story generation by defining 19 fine-grained error subtypes.
• The benchmark employs a four-stage automated pipeline, ConStory-Checker, to extract, classify, and report inconsistencies using token entropy and other metrics.
• Empirical findings show that error density rises with story length and that models like GPT-5-Reasoning perform best, underscoring the need for improved state tracking.

Searching arXiv for the specified benchmark and closely related work to ground the article in current papers. ConStory-Bench is a benchmark designed to evaluate narrative consistency in long-form story generation. Paired with ConStory-Checker, it forms a framework for measuring and understanding consistency failures in ultra-long narrative generation by LLMs, with 2,000 prompts, four task scenarios, and a five-dimension taxonomy comprising 19 fine-grained subtypes of consistency error (Li et al., 6 Mar 2026).

1. Definition, scope, and disambiguation

ConStory-Bench addresses a specific gap in long-form narrative evaluation: existing story generation benchmarks focus mainly on plot quality and fluency, leaving consistency errors largely unexplored. Its central object of study is not stylistic preference or general quality alone, but contradiction against a story’s own established facts, character traits, and world rules across narratives intended to span 8,000–10,000 words (Li et al., 6 Mar 2026).

The benchmark is distinct from the earlier video-generation benchmark "StoryBench: A Multifaceted Benchmark for Continuous Story Visualization" (Bugliarello et al., 2023). That benchmark targets continuous story visualization and evaluates action execution, story continuation, and story generation for text-to-video systems, whereas ConStory-Bench targets textual narrative consistency in long-form LLM generation. The similarity in naming can create ambiguity, but the two benchmarks operate on different modalities, task definitions, and evaluation objectives (Bugliarello et al., 2023).

A plausible implication is that ConStory-Bench should be understood as a benchmark for intra-narrative logical and factual stability in long outputs, rather than as a general-purpose story-quality leaderboard.

2. Dataset design and task scenarios

ConStory-Bench comprises 2,000 high-quality prompts, each designed to elicit an 8,000–10,000-word narrative. Seed stories were drawn from seven public corpora: LongBench, LongBench_Write, LongLamp, TellMeAStory, WritingBench, WritingPrompts, and WikiPlots. The source material covers both open-ended scenarios and fully fleshed narratives, after which each seed story was assigned to one of four task types based on its structure (Li et al., 6 Mar 2026).

Task type	Prompts	Share
Generation	751	37.5%
Continuation	432	21.6%
Expansion	422	21.1%
Completion	395	19.8%

The four task types are defined by their conditioning structure. Generation provides a minimal plot setup and asks for a free-form narrative. Continuation extends an opening fragment into a complete story. Expansion elaborates a concise yet coherent outline. Completion fills in the middle of a story given its beginning and ending. These task distinctions matter because they modulate how much prior structure the model can rely on before long-range consistency must be maintained (Li et al., 6 Mar 2026).

To convert seed texts into model-ready prompts, the benchmark uses o4-mini to rewrite each story into the target task format while constraining the expected generation to 8,000–10,000 words. MinHash-based deduplication removes near-duplicate prompts, and trivial or low-quality cases are filtered via automated heuristics and manual inspection. Across model outputs, average story lengths varied by system, from around 1,241 words for GPT-4o-1120 up to 13,393 words for LongWriter-Zero. In general, proprietary models such as GPT-5-Reasoning and Claude-Sonnet-4.5 produced 6,000+-word stories over 90% of the time, while some open-source and capability-enhanced systems favored shorter outputs (Li et al., 6 Mar 2026).

3. Consistency taxonomy

The benchmark organizes consistency failures into five top-level categories and 19 fine-grained subtypes. Each subtype is intended to capture a distinct form of contradiction rather than a vague notion of narrative weakness (Li et al., 6 Mar 2026).

Timeline & Plot Logic includes six subtypes: Absolute Time Contradictions, Duration Contradictions, Simultaneity Contradictions, Causeless Effects, Causal Logic Violations, and Abandoned Plot Elements. These errors concern temporal reference, event ordering, causal structure, and unresolved threads.

Characterization includes Memory Contradictions, Knowledge Contradictions, Skill/Power Fluctuations, and Forgotten Abilities. These failures arise when a character’s recollection, background knowledge, or operative capabilities conflict with prior narrative establishment.

World-building & Setting includes Core Rules Violations, Social Norms Violations, and Geographical Contradictions. These subtypes target inconsistency in physical or magical law, sociocultural systems, and the spatial properties of the story world.

Factual Detail Consistency includes Appearance Mismatches, Nomenclature Confusions, and Quantitative Mismatches. These are localized but often highly salient contradictions in descriptive attributes, naming, and numerically specified facts.

Narrative Style includes Perspective Confusions, Tone Inconsistencies, and Style Shifts. Unlike the other categories, these do not primarily concern diegetic facts; they concern instability in viewpoint, emotional register, and linguistic realization (Li et al., 6 Mar 2026).

The taxonomy is notable because it separates logically distinct failure modes that are often collapsed into a generic label such as “hallucination” or “incoherence.” This suggests that mitigation strategies may need to be subtype-specific rather than optimized against a single aggregate consistency score.

4. ConStory-Checker

ConStory-Checker is the automated pipeline used to detect and classify inconsistencies at scale. It is a four-stage LLM-as-judge system, and all stages use o4-mini for a balance of speed and accuracy (Li et al., 6 Mar 2026).

Stage 1: Category-Guided Extraction runs tailored extraction prompts for each of the five dimensions and highlights passages likely to contain contradictions. For the Timeline category, for example, the prompt asks for sentences that mention a date or season.

Stage 2: Contradiction Pairing compares extracted spans pairwise. Drawing on the CheckEval and ProxyQA protocols, the model judges whether each pair is “Consistent” or “Contradictory,” with the explicit goal of reducing false positives and isolating genuine conflicts.

Stage 3: Evidence Chains produces, for every detected contradiction, a triplet of fields: Reasoning, Evidence, and Conclusion. Evidence contains exact quotes with character-level offsets, including fact_quote, contradiction_quote, and locations. Conclusion assigns the final error subtype, such as absolute_time_error.

Stage 4: JSON Reports serializes quotations, offsets, contradiction pairings, and error labels into a standardized JSON schema for downstream analysis and human inspection (Li et al., 6 Mar 2026).

Validation was carried out on a diagnostic set of 200 stories with 1,000 injected errors. Against human expert annotations, ConStory-Checker achieved 55.0% recall at 88.4% precision, with $F1=0.678$, substantially outperforming human recall at 17.1% and $F1=0.229$. The comparison is specific to the diagnostic detection setting; a plausible implication is that the pipeline is better suited for large-scale screening than unaided manual review when contradiction evidence must be surfaced explicitly (Li et al., 6 Mar 2026).

5. Evaluation methodology and metrics

The evaluation is organized around five research questions: how well current LLMs maintain narrative coherence in ultra-long generation and whether they show similar error-type distributions; how consistency errors scale with output length; what factors, particularly model uncertainty, signal the emergence of errors; whether error types co-occur systematically; and how errors are distributed by position within the narrative (Li et al., 6 Mar 2026).

The experimental suite includes proprietary systems such as GPT-5-Reasoning, Gemini-2.5-Pro, and Claude-Sonnet-4.5; open-source systems such as Qwen3-32B and GLM-4.6; capability-enhanced systems such as LongWriter-Zero; and agent-enhanced systems such as SuperWriter and DOME (Li et al., 6 Mar 2026).

Two primary metrics quantify consistency. Consistency Error Density (CED) normalizes errors by length as errors per 10,000 words; lower is better. Group Relative Rank (GRR) controls for prompt difficulty and length-completeness trade-offs by ranking outputs per prompt with a length-aware quality score; lower GRR indicates relatively better performance across prompts (Li et al., 6 Mar 2026).

To study uncertainty, the benchmark computes token-level Shannon entropy,

$H(P_t)=-\sum_{i=1}^K p_i \log_2 p_i,$

and also examines token probability and perplexity in the appendix. For co-occurrence analysis, it computes Pearson correlations among the five category-level error densities across all outputs. For positional analysis, it records the normalized fact position, contradiction position, and average gap between them (Li et al., 6 Mar 2026).

The methodological structure is important because it separates three issues that are often confounded in long-form evaluation: total error burden, the effect of output length, and the internal structure of errors once they occur.

6. Empirical findings and research significance

On overall performance, GPT-5-Reasoning leads with the lowest overall CED at 0.113 errors per 10K words and the best GRR at 3.05. Gemini-2.5-Pro follows with $\text{CED}=0.305$ and $\text{GRR} \approx 7.8$, and Claude-Sonnet-4.5 with $\text{CED}=0.520$ and $\text{GRR}=4.90$. Among open-source models, GLM-4.6 at $\text{CED}=0.528$ and Qwen3-32B at $\text{CED}=0.537$ approach proprietary performance. Capability-enhanced LongWriter-Zero at $\text{CED}=0.669$ and agent-enhanced SuperWriter at $F1=0.229$0 are also competitive (Li et al., 6 Mar 2026).

Across all systems, Factual Detail Consistency, with average CED approximately 0.18, and Timeline & Plot Logic, with average CED approximately 0.11, dominate error profiles. Characterization, World-building, and Narrative Style errors occur far less frequently, each below 0.035. Generation tasks consistently yield higher CED than Continuation, Expansion, and Completion, indicating that open-ended creation without prior context is most prone to contradictions (Li et al., 6 Mar 2026).

Length dynamics are pronounced. Error counts grew approximately linearly with length across models. DeepSeek-V3.2-Exp shows an almost perfect correlation of $F1=0.229$1 between story length and error count, while Claude-Sonnet-4.5 has a moderate correlation of $F1=0.229$2. This indicates that longer stories accumulate more inconsistencies, but that different length preferences create distinct trade-off patterns (Li et al., 6 Mar 2026).

Uncertainty signals are also systematic. Across two open-source models, error segments exhibited substantially higher Shannon entropy than the story-wide baseline: Qwen3-30B-A3B shows a 12.03% increase and Qwen3-4B-Instruct a 19.24% increase. Complementary metrics show token probability down by 5–8% and perplexity up by 2.5–5.5%. The paper states that this suggests entropy and related measures can serve as an early-warning signal to trigger on-the-fly self-checks (Li et al., 6 Mar 2026).

Error types do not behave independently. Factual Detail Consistency functions as a central hub, correlating strongly with Characterization at $F1=0.229$3, World-building at $F1=0.229$4, and Timeline Logic at $F1=0.229$5. Narrative Style errors exhibit near-zero correlations with all other categories, indicating distinct failure mechanisms. Positional analysis shows that facts are typically introduced in the first 20–30% of the narrative, while contradictions surface around 40–60%. Absolute Time Contradictions have average fact position 22.6%, average contradiction position 48.9%, and average gap 29.7%; Geographical Contradictions show an even larger gap of 31.0%. Perspective Confusions occur locally, with average gap 4.7%, confirming that stylistic shifts are short-range failures (Li et al., 6 Mar 2026).

Taken together, these results identify factual tracking and temporal reasoning as the primary consistency hurdles in ultra-long story generation. A plausible implication is that future mitigation will likely require mechanisms for explicit state tracking, long-range retrieval over prior narrative commitments, and uncertainty-aware self-verification, especially in the middle portions of long narratives where contradictions most often surface.