ConStory-Checker: Narrative Consistency Evaluation

• ConStory-Checker is an automated LLM-as-a-judge pipeline that rigorously detects narrative consistency errors using explicit textual evidence.
• It employs a 4-stage process—extraction, pairing, evidence chaining, and JSON reporting—to isolate and classify contradictions across narrative dimensions.
• Benchmarked on over 2,000 prompts, it improves error discovery by 3.2× over human annotators, highlighting its precision in long-form story evaluations.

ConStory-Checker is an automated LLM-as-a-judge pipeline for detecting narrative consistency errors in long-form story generation. Introduced together with ConStory-Bench in “Lost in Stories: Consistency Bugs in Long Story Generation by LLMs,” it is designed to analyze stories in the 8,000–10,000 word range, identify contradictions across five narrative dimensions, and ground each judgment in explicit textual evidence rather than only assigning a scalar score. Within that framework, ConStory-Bench provides the benchmark and prompt set, while ConStory-Checker provides the benchmark’s core evaluator and structured contradiction reports (Li et al., 6 Mar 2026).

1. Benchmark setting and problem scope

ConStory-Checker is tightly coupled to ConStory-Bench, a benchmark containing 2,000 prompts across four task types: Generation, Continuation, Expansion, and Completion. The task distribution is reported as 751 Generation prompts, 432 Continuation prompts, 422 Expansion prompts, and 395 Completion prompts. The prompts were derived from seven public corpora—LongBench, LongBench_Write, LongLamp, TellMeAStory / Storium, WritingBench, WritingPrompts, and WikiPlots—then rewritten with o4-mini, deduplicated with MinHash, and filtered manually and with heuristics (Li et al., 6 Mar 2026).

The benchmark is motivated by a specific failure mode of long-context LLMs: as narratives grow longer, models often “forget their own story.” In the formulation used by ConStory-Checker, a consistency bug is a contradiction between parts of the same generated story, where later content conflicts with earlier established facts, logic, or conventions. The framework treats earlier narrative content as ground truth when evaluating later contradiction. That asymmetry is central: ConStory-Checker is not performing symmetric similarity scoring, but directed contradiction detection relative to an already established narrative baseline (Li et al., 6 Mar 2026).

This design places ConStory-Checker in the niche left open by prior long-form evaluation practices that emphasize plot quality, fluency, or holistic writing quality without isolating cross-context contradiction patterns. A plausible implication is that the system’s main target is not general literary quality, but the more specific question of whether a long narrative preserves its own internal commitments over thousands of tokens.

2. Consistency ontology and error model

ConStory-Checker operationalizes narrative consistency through 5 coarse categories and 19 fine-grained subtypes. The five top-level dimensions are Timeline & Plot Logic, Characterization, World-building & Setting, Factual & Detail Consistency, and Narrative & Style (Li et al., 6 Mar 2026).

The taxonomy covers the following subtypes:

• Timeline & Plot Logic: Absolute Time Contradictions, Duration Contradictions, Simultaneity Contradictions, Causeless Effects, Causal Logic Violations, and Abandoned Plot Elements.
• Characterization: Memory Contradictions, Knowledge Contradictions, Skill Fluctuations, and Forgotten Abilities.
• World-building & Setting: Core Rules Violations, Social Norms Violations, and Geographical Contradictions.
• Factual & Detail Consistency: Appearance Mismatches, Nomenclature Confusions, and Quantitative Mismatches.
• Narrative & Style: Perspective Confusions, Tone Inconsistencies, and Style Shifts (Li et al., 6 Mar 2026).

The paper gives representative examples for several of these classes. For Absolute Time Contradictions, a prompt template contrasts a “scorching July afternoon” with subsequent heavy snow. For Memory Contradictions, one template uses “I’ve never seen this woman...” despite earlier interaction. For Geographical Contradictions, the appendix contrasts an opera house described as ruined and forgotten with a later description of the same place as active and full of life. For Appearance Mismatches, one example changes a character’s eye color from blue to emerald. For Perspective Confusions, the example is a shift between first-person and third-person in the same paragraph (Li et al., 6 Mar 2026).

The checker’s notion of a consistency bug is therefore broader than factual contradiction in the narrow sense. It includes temporal order, causal setup, memory continuity, world rules, naming, numerical details, and stylistic continuity. At the same time, the paper states an important boundary condition: the framework uses a binary consistency judgment—consistent versus contradictory—and does not distinguish intentional ambiguity or surprise reveals from genuine errors. This means the ontology is fine-grained, but the final decision rule is deliberately coarse (Li et al., 6 Mar 2026).

3. Pipeline architecture and evidence representation

ConStory-Checker is described as a multi-stage contradiction detection pipeline. The main text presents it as a conceptual 4-stage pipeline, and the appendix provides category-specific prompt specifications and JSON schemas (Li et al., 6 Mar 2026).

The first stage is category-guided extraction. The story is scanned with category-specific judge prompts for each of the five top-level dimensions. These prompts instruct the judge model to extract candidate contradiction-prone spans, using subtype-specific “look for” guidance. The intermediate representation includes standardized fields such as fact_quote, location, contradiction_pair, contradiction_location, error_element, error_category, and context (Li et al., 6 Mar 2026).

The second stage is contradiction pairing. Extracted spans are then compared pairwise and classified as either Consistent or Contradictory. The paper explicitly says this stage follows ideas from CheckEval and ProxyQA and is intended to reduce false positives by separating genuinely contradictory pairs from merely suspicious text spans. Functionally, this stage acts as a pairwise contradiction classifier, although the paper does not explicitly label it as NLI (Li et al., 6 Mar 2026).

The third stage constructs evidence chains. For each confirmed contradiction, the pipeline records a structured triple consisting of Reasoning, Evidence, and Conclusion: why the passages contradict, which quotations and positions support the judgment, and which error type is assigned. This is one of the system’s defining features, because ConStory-Checker does not only emit category labels; it produces auditable contradiction cases with exact references (Li et al., 6 Mar 2026).

The fourth stage writes standardized JSON reports. The final output contains quotations, positions, contradiction pairings, error categories, and explanations. The paper states that judgments are anchored to precise character-level offsets; appendix prompt examples also use chapter/paragraph references. The evaluator model used for the pipeline is o4-mini, chosen to balance accuracy and efficiency. The paper does not specify context-window management, chunk sizes, batching strategy, decoding parameters for ConStory-Checker itself, thresholds, score calibration procedures, post-hoc deduplication heuristics, or pseudocode for the checker (Li et al., 6 Mar 2026).

A plausible implication is that the core technical novelty lies less in trainable architecture than in prompted extraction, pairwise verification, and evidence-grounded reporting.

4. Scoring framework and checker validation

ConStory-Checker feeds directly into the benchmark’s scoring framework. The paper defines Consistency Error Density (CED) to normalize error counts by story length, and Group Relative Rank (GRR) to rank models within prompt groups using a quality score that balances output length and detected error count; for both metrics, lower is better. It also defines a positional statistic, Avg Gap, for the normalized distance between the position where a fact is established and the position where its contradiction appears. In a separate analysis, token-level entropy, token probability, and perplexity are used to compare error-bearing segments with whole-story baselines, but that entropy analysis is explicitly described as not part of ConStory-Checker’s inference pipeline (Li et al., 6 Mar 2026).

The checker itself is validated on a diagnostic dataset with injected errors. The dataset contains 200 stories generated using Qwen3-235B-A22B-Thinking, with 1,000 injected inconsistencies distributed across the five major dimensions, effectively 200 per major category. Human annotation was performed by two professional web novel writers, paid \$1.00 per story, who completed the work within two days. They were asked to study the five-category taxonomy and subtype definitions, then annotate fact quote, contradicting quote, error category and subtype, and explanation. The paper does not report inter-annotator agreement statistics such as Cohen’s kappa (Li et al., 6 Mar 2026).

On this validation set, ConStory-Checker achieves Precision = 0.884, Recall = 0.550, and F1 = 0.678. Human experts, averaged, achieve Precision = 0.660, Recall = 0.139, and F1 = 0.229. The appendix prose also mentions “Overall F1=0.281” for humans, but that conflicts with the table value of 0.229; the table is the more explicit numerical source. The checker detects 550/1000 injected errors, versus 138.5 true positives on average for human annotators, summarized in the paper as a 3.2× improvement in error discovery rate (Li et al., 6 Mar 2026).

Per-category performance further clarifies the operating profile. ConStory-Checker reports Recall 0.605, Precision 0.960, F1 0.742 for Character Consistency; 0.625, 0.845, 0.718 for Factual Accuracy; 0.350, 0.921, 0.507 for Narrative Coherence; 0.600, 0.816, 0.692 for Temporal Logic; and 0.570, 0.912, 0.702 for World Consistency. The lowest F1 is therefore in Narrative Coherence, which the paper later identifies as a harder class (Li et al., 6 Mar 2026).

5. Benchmark findings enabled by ConStory-Checker

ConStory-Checker is the mechanism used to answer the benchmark’s five research questions. Under RQ1, it reveals substantial variation across models. The best reported model is GPT-5-Reasoning with CED 0.113 and GRR 3.05 in the table, although the main text gives best GRR as 2.80. Gemini-2.5-Pro has CED 0.305, and Claude-Sonnet-4.5 has CED 0.520 with GRR 4.9 in the table, while the prose gives 4.54. Among open-source models, GLM-4.6 has CED 0.528 and Qwen3-32B has CED 0.537. The paper identifies Factual & Detail Consistency and Timeline & Plot Logic as the dominant failure modes across models, and reports that Generation tasks are consistently hardest (Li et al., 6 Mar 2026).

Under RQ2, ConStory-Checker’s error counts show that errors increase approximately linearly with output length. The reported example correlations are $r = 0.478$ for Claude-Sonnet-4.5 and $r = 0.973$ for DeepSeek-V3.2-Exp. This suggests that output length is not merely a nuisance variable but a major driver of inconsistency opportunity (Li et al., 6 Mar 2026).

Under RQ3, ConStory-Checker is used to locate error-bearing segments, which are then analyzed via token-level uncertainty. For Qwen3-4B-Instruct-2507, error segments have +19.24% higher entropy; for Qwen3-30B-A3B-Instruct-2507, the increase is +12.03%. Appendix statistics further report lower token probability and higher perplexity in error segments. The paper interprets this as evidence that consistency errors tend to emerge in regions of greater uncertainty, making entropy a plausible trigger for verification or self-check routines (Li et al., 6 Mar 2026).

Under RQ4, ConStory-Checker’s categorical outputs support co-occurrence analysis. Factual & Detail Consistency acts as a central hub, with Pearson correlations of $r = 0.304$ with Characterization, $r = 0.255$ with World-building & Setting, and $r = 0.176$ with Timeline & Plot Logic. By contrast, Narrative & Style errors show near-zero correlation with other categories. The paper interprets this as indicating that factual mistakes often co-occur with broader coherence problems, whereas stylistic inconsistency may arise from a different mechanism (Li et al., 6 Mar 2026).

Under RQ5, the positional metadata recorded by ConStory-Checker show that contradictions tend to occur in the middle region of stories, especially around 40–60% of narrative length, while facts are usually established earlier, around 15–30%. Representative subtype gaps include Geographical Contradictions with Avg Gap 31.0%, Absolute Time Contradictions with 29.7%, Memory Contradictions with 25.4%, and Perspective Confusions with 4.7%. The paper connects this pattern to “lost in the middle”-type long-context issues, though it presents that link as interpretive rather than formally tested (Li et al., 6 Mar 2026).

6. Methodological position, related work, and limitations

ConStory-Checker belongs to a broader family of consistency-checking systems, but its operational target is unusually specific: evidence-grounded contradiction detection inside long generated stories. That distinguishes it from engineering consistency checkers over heterogeneous artifacts, such as the DesignSpace-based team-oriented framework for requirements, specifications, and code, which is organized around property-based artifact representation, grouped work areas, and global consistency rules rather than narrative contradiction extraction (Tröls et al., 2021). It also differs from formal satisfiability-based checking of functional requirements, where consistency is defined as whether the conjunction of temporal-logic encodings is satisfiable (Vuotto, 2018).

It should also be distinguished from evaluation frameworks that use “consistency” in other senses. ConsistencyChecker measures end-to-end preservation under repeated reversible transformations in translation and programming via a self-consistency tree and depth-indexed scores such as $C_3(\mathcal{F})$, but it does not detect contradiction cases inside a single long narrative (Hong et al., 14 Jun 2025). WeCheck treats factual consistency as premise–hypothesis classification under weak supervision and reports 84.8 ROC AUC on TRUE, but it is oriented toward source-conditioned generated text rather than cross-context story contradictions (Wu et al., 2022). ConFactCheck probes extracted key facts in generated text by asking targeted factual questions and testing alignment and token-distribution confidence, which is closer in spirit to fact-level instability detection, though it is aimed at hallucination detection rather than narrative taxonomy-driven contradiction reports (Gupta et al., 15 Nov 2025). BiCon-Gate contributes a semantics-aware gate for accepting or rejecting dialogue rewrites based on bidirectional NLI and cosine similarity, which is useful for local rewrite faithfulness, but not for global long-story consistency (Park et al., 15 Apr 2026). On the generation side, “Consistency and Coherency Enhanced Story Generation” improves plot control through an outline, coreference supervision, and discourse relation modeling, but that work focuses on reducing inconsistency during generation rather than auditing completed stories with evidence-grounded contradiction chains (Wang et al., 2020).

The explicit limitations of ConStory-Checker are correspondingly important. The benchmark is restricted to English fiction / Western narrative conventions only. The checker uses binary consistency modeling, so some apparent contradictions may actually be intentional devices such as surprise endings or delayed revelations. Validation uses synthetically injected errors, which may not fully match naturally occurring inconsistencies. The system depends on an LLM judge—specifically o4-mini—so judgments may inherit model biases or failure modes. The paper does not report inter-annotator agreement. Finally, recall remains 0.550, which means many errors are still undetected, and Narrative Coherence is the weakest major category by F1 (Li et al., 6 Mar 2026).

Taken together, these properties position ConStory-Checker as a benchmark-centered, taxonomy-driven evaluator: it is scalable, auditable, and fine-grained, but it is not a full formal semantics of narrative consistency. A plausible implication is that its greatest value lies in making long-story inconsistency measurable at scale, while leaving open deeper questions about intentional ambiguity, world modeling, and long-range causal semantics.