---
title: 'ConStory-Checker: Narrative Consistency Evaluation'
url: https://www.emergentmind.com/topics/constory-checker
type: topic
---

# ConStory-Checker: Narrative Consistency Evaluation

ConStory-Checker is an automated **LLM-as-a-judge pipeline** for detecting narrative consistency errors in long-form story generation. Introduced together with **ConStory-Bench** in “Lost in Stories: Consistency Bugs in Long Story Generation by LLMs,” it is designed to analyze stories in the **8,000–10,000 word** range, identify contradictions across **five narrative dimensions**, and ground each judgment in **explicit textual evidence** rather than only assigning a scalar score. Within that framework, ConStory-Bench provides the benchmark and prompt set, while ConStory-Checker provides the benchmark’s core evaluator and structured contradiction reports [2603.05890].

## 1. Benchmark setting and problem scope

ConStory-Checker is tightly coupled to **ConStory-Bench**, a benchmark containing **2,000 prompts** across **four task types**: **Generation**, **Continuation**, **Expansion**, and **Completion**. The task distribution is reported as **751** Generation prompts, **432** Continuation prompts, **422** Expansion prompts, and **395** Completion prompts. The prompts were derived from seven public corpora—**LongBench**, **LongBench_Write**, **LongLamp**, **TellMeAStory / Storium**, **WritingBench**, **WritingPrompts**, and **WikiPlots**—then rewritten with **o4-mini**, deduplicated with **MinHash**, and filtered manually and with heuristics [2603.05890].

The benchmark is motivated by a specific failure mode of long-context language models: as narratives grow longer, models often “forget their own story.” In the formulation used by ConStory-Checker, a **consistency bug** is a contradiction between parts of the same generated story, where later content conflicts with earlier established facts, logic, or conventions. The framework treats **earlier narrative content as ground truth** when evaluating later contradiction. That asymmetry is central: ConStory-Checker is not performing symmetric similarity scoring, but directed contradiction detection relative to an already established narrative baseline [2603.05890].

This design places ConStory-Checker in the niche left open by prior long-form evaluation practices that emphasize **plot quality**, **fluency**, or **holistic writing quality** without isolating cross-context contradiction patterns. A plausible implication is that the system’s main target is not general literary quality, but the more specific question of whether a long narrative preserves its own internal commitments over thousands of tokens.

## 2. Consistency ontology and error model

ConStory-Checker operationalizes narrative consistency through **5 coarse categories** and **19 fine-grained subtypes**. The five top-level dimensions are **Timeline & Plot Logic**, **Characterization**, **World-building & Setting**, **Factual & Detail Consistency**, and **Narrative & Style** [2603.05890].

The taxonomy covers the following subtypes:

- **Timeline & Plot Logic**: **Absolute Time Contradictions**, **Duration Contradictions**, **Simultaneity Contradictions**, **Causeless Effects**, **Causal Logic Violations**, and **Abandoned Plot Elements**.
- **Characterization**: **Memory Contradictions**, **Knowledge Contradictions**, **Skill Fluctuations**, and **Forgotten Abilities**.
- **World-building & Setting**: **Core Rules Violations**, **Social Norms Violations**, and **Geographical Contradictions**.
- **Factual & Detail Consistency**: **Appearance Mismatches**, **Nomenclature Confusions**, and **Quantitative Mismatches**.
- **Narrative & Style**: **Perspective Confusions**, **Tone Inconsistencies**, and **Style Shifts** [2603.05890].

The paper gives representative examples for several of these classes. For **Absolute Time Contradictions**, a prompt template contrasts a “scorching July afternoon” with subsequent heavy snow. For **Memory Contradictions**, one template uses “I’ve never seen this woman...” despite earlier interaction. For **Geographical Contradictions**, the appendix contrasts an opera house described as ruined and forgotten with a later description of the same place as active and full of life. For **Appearance Mismatches**, one example changes a character’s eye color from blue to emerald. For **Perspective Confusions**, the example is a shift between first-person and third-person in the same paragraph [2603.05890].

The checker’s notion of a consistency bug is therefore broader than factual contradiction in the narrow sense. It includes temporal order, causal setup, memory continuity, world rules, naming, numerical details, and stylistic continuity. At the same time, the paper states an important boundary condition: the framework uses a **binary consistency judgment**—**consistent** versus **contradictory**—and does not distinguish intentional ambiguity or surprise reveals from genuine errors. This means the ontology is fine-grained, but the final decision rule is deliberately coarse [2603.05890].

## 3. Pipeline architecture and evidence representation

ConStory-Checker is described as a **multi-stage contradiction detection pipeline**. The main text presents it as a conceptual **4-stage pipeline**, and the appendix provides category-specific prompt specifications and JSON schemas [2603.05890].

The first stage is **category-guided extraction**. The story is scanned with category-specific judge prompts for each of the five top-level dimensions. These prompts instruct the judge model to extract candidate contradiction-prone spans, using subtype-specific “look for” guidance. The intermediate representation includes standardized fields such as `fact_quote`, `location`, `contradiction_pair`, `contradiction_location`, `error_element`, `error_category`, and `context` [2603.05890].

The second stage is **contradiction pairing**. Extracted spans are then compared pairwise and classified as either **Consistent** or **Contradictory**. The paper explicitly says this stage follows ideas from **CheckEval** and **ProxyQA** and is intended to reduce false positives by separating genuinely contradictory pairs from merely suspicious text spans. Functionally, this stage acts as a pairwise contradiction classifier, although the paper does not explicitly label it as NLI [2603.05890].

The third stage constructs **evidence chains**. For each confirmed contradiction, the pipeline records a structured triple consisting of **Reasoning**, **Evidence**, and **Conclusion**: why the passages contradict, which quotations and positions support the judgment, and which error type is assigned. This is one of the system’s defining features, because ConStory-Checker does not only emit category labels; it produces **auditable contradiction cases** with exact references [2603.05890].

The fourth stage writes **standardized JSON reports**. The final output contains quotations, positions, contradiction pairings, error categories, and explanations. The paper states that judgments are anchored to **precise character-level offsets**; appendix prompt examples also use chapter/paragraph references. The evaluator model used for the pipeline is **o4-mini**, chosen to balance accuracy and efficiency. The paper does **not** specify context-window management, chunk sizes, batching strategy, decoding parameters for ConStory-Checker itself, thresholds, score calibration procedures, post-hoc deduplication heuristics, or pseudocode for the checker [2603.05890].

A plausible implication is that the core technical novelty lies less in trainable architecture than in **prompted extraction, pairwise verification, and evidence-grounded reporting**.

## 4. Scoring framework and checker validation

ConStory-Checker feeds directly into the benchmark’s scoring framework. The paper defines **Consistency Error Density (CED)** to normalize error counts by story length, and **Group Relative Rank (GRR)** to rank models within prompt groups using a quality score that balances output length and detected error count; for both metrics, **lower is better**. It also defines a positional statistic, **Avg Gap**, for the normalized distance between the position where a fact is established and the position where its contradiction appears. In a separate analysis, token-level **entropy**, token probability, and **perplexity** are used to compare error-bearing segments with whole-story baselines, but that entropy analysis is explicitly described as **not part of ConStory-Checker’s inference pipeline** [2603.05890].

The checker itself is validated on a **diagnostic dataset with injected errors**. The dataset contains **200 stories** generated using **Qwen3-235B-A22B-Thinking**, with **1,000** injected inconsistencies distributed across the five major dimensions, effectively **200 per major category**. Human annotation was performed by **two professional web novel writers**, paid **\$1.00 per story**, who completed the work within **two days**. They were asked to study the five-category taxonomy and subtype definitions, then annotate fact quote, contradicting quote, error category and subtype, and explanation. The paper does **not** report inter-annotator agreement statistics such as Cohen’s kappa [2603.05890].

On this validation set, ConStory-Checker achieves **Precision = 0.884**, **Recall = 0.550**, and **F1 = 0.678**. Human experts, averaged, achieve **Precision = 0.660**, **Recall = 0.139**, and **F1 = 0.229**. The appendix prose also mentions “Overall F1=0.281” for humans, but that conflicts with the table value of **0.229**; the table is the more explicit numerical source. The checker detects **550/1000 injected errors**, versus **138.5 true positives on average** for human annotators, summarized in the paper as a **3.2× improvement in error discovery rate** [2603.05890].

Per-category performance further clarifies the operating profile. ConStory-Checker reports **Recall 0.605, Precision 0.960, F1 0.742** for **Character Consistency**; **0.625, 0.845, 0.718** for **Factual Accuracy**; **0.350, 0.921, 0.507** for **Narrative Coherence**; **0.600, 0.816, 0.692** for **Temporal Logic**; and **0.570, 0.912, 0.702** for **World Consistency**. The lowest F1 is therefore in **Narrative Coherence**, which the paper later identifies as a harder class [2603.05890].

## 5. Benchmark findings enabled by ConStory-Checker

ConStory-Checker is the mechanism used to answer the benchmark’s five research questions. Under **RQ1**, it reveals substantial variation across models. The best reported model is **GPT-5-Reasoning** with **CED 0.113** and **GRR 3.05** in the table, although the main text gives best GRR as **2.80**. **Gemini-2.5-Pro** has **CED 0.305**, and **Claude-Sonnet-4.5** has **CED 0.520** with **GRR 4.9** in the table, while the prose gives **4.54**. Among open-source models, **GLM-4.6** has **CED 0.528** and **Qwen3-32B** has **CED 0.537**. The paper identifies **Factual & Detail Consistency** and **Timeline & Plot Logic** as the dominant failure modes across models, and reports that **Generation** tasks are consistently hardest [2603.05890].

Under **RQ2**, ConStory-Checker’s error counts show that errors increase **approximately linearly** with output length. The reported example correlations are **\(r = 0.478\)** for **Claude-Sonnet-4.5** and **\(r = 0.973\)** for **DeepSeek-V3.2-Exp**. This suggests that output length is not merely a nuisance variable but a major driver of inconsistency opportunity [2603.05890].

Under **RQ3**, ConStory-Checker is used to locate error-bearing segments, which are then analyzed via token-level uncertainty. For **Qwen3-4B-Instruct-2507**, error segments have **+19.24%** higher entropy; for **Qwen3-30B-A3B-Instruct-2507**, the increase is **+12.03%**. Appendix statistics further report lower token probability and higher perplexity in error segments. The paper interprets this as evidence that consistency errors tend to emerge in regions of **greater uncertainty**, making entropy a plausible trigger for verification or self-check routines [2603.05890].

Under **RQ4**, ConStory-Checker’s categorical outputs support co-occurrence analysis. **Factual & Detail Consistency** acts as a central hub, with Pearson correlations of **\(r = 0.304\)** with **Characterization**, **\(r = 0.255\)** with **World-building & Setting**, and **\(r = 0.176\)** with **Timeline & Plot Logic**. By contrast, **Narrative & Style** errors show near-zero correlation with other categories. The paper interprets this as indicating that factual mistakes often co-occur with broader coherence problems, whereas stylistic inconsistency may arise from a different mechanism [2603.05890].

Under **RQ5**, the positional metadata recorded by ConStory-Checker show that contradictions tend to occur in the **middle region of stories**, especially around **40–60%** of narrative length, while facts are usually established earlier, around **15–30%**. Representative subtype gaps include **Geographical Contradictions** with **Avg Gap 31.0%**, **Absolute Time Contradictions** with **29.7%**, **Memory Contradictions** with **25.4%**, and **Perspective Confusions** with **4.7%**. The paper connects this pattern to “lost in the middle”-type long-context issues, though it presents that link as interpretive rather than formally tested [2603.05890].

## 6. Methodological position, related work, and limitations

ConStory-Checker belongs to a broader family of consistency-checking systems, but its operational target is unusually specific: **evidence-grounded contradiction detection inside long generated stories**. That distinguishes it from engineering consistency checkers over heterogeneous artifacts, such as the DesignSpace-based team-oriented framework for requirements, specifications, and code, which is organized around **property-based artifact representation**, **grouped work areas**, and **global consistency rules** rather than narrative contradiction extraction [2103.14860]. It also differs from formal satisfiability-based checking of functional requirements, where consistency is defined as whether the conjunction of temporal-logic encodings is satisfiable [1804.10486].

It should also be distinguished from evaluation frameworks that use “consistency” in other senses. **ConsistencyChecker** measures end-to-end preservation under repeated reversible transformations in translation and programming via a self-consistency tree and depth-indexed scores such as \(C_3(\mathcal{F})\), but it does not detect contradiction cases inside a single long narrative [2506.12376]. **WeCheck** treats factual consistency as premise–hypothesis classification under weak supervision and reports **84.8 ROC AUC** on TRUE, but it is oriented toward source-conditioned generated text rather than cross-context story contradictions [2212.10057]. **ConFactCheck** probes extracted key facts in generated text by asking targeted factual questions and testing alignment and token-distribution confidence, which is closer in spirit to fact-level instability detection, though it is aimed at hallucination detection rather than narrative taxonomy-driven contradiction reports [2511.12236]. **BiCon-Gate** contributes a semantics-aware gate for accepting or rejecting dialogue rewrites based on bidirectional NLI and cosine similarity, which is useful for local rewrite faithfulness, but not for global long-story consistency [2604.14389]. On the generation side, “Consistency and Coherency Enhanced Story Generation” improves plot control through an outline, coreference supervision, and discourse relation modeling, but that work focuses on reducing inconsistency during generation rather than auditing completed stories with evidence-grounded contradiction chains [2010.08822].

The explicit limitations of ConStory-Checker are correspondingly important. The benchmark is restricted to **English fiction / Western narrative conventions only**. The checker uses **binary consistency modeling**, so some apparent contradictions may actually be intentional devices such as surprise endings or delayed revelations. Validation uses **synthetically injected errors**, which may not fully match naturally occurring inconsistencies. The system depends on an **LLM judge**—specifically **o4-mini**—so judgments may inherit model biases or failure modes. The paper does not report inter-annotator agreement. Finally, recall remains **0.550**, which means many errors are still undetected, and **Narrative Coherence** is the weakest major category by F1 [2603.05890].

Taken together, these properties position ConStory-Checker as a benchmark-centered, taxonomy-driven evaluator: it is **scalable**, **auditable**, and **fine-grained**, but it is not a full formal semantics of narrative consistency. A plausible implication is that its greatest value lies in making long-story inconsistency measurable at scale, while leaving open deeper questions about intentional ambiguity, world modeling, and long-range causal semantics.

Source: https://www.emergentmind.com/topics/constory-checker