LongStoryEval: Evaluating Book-Length Narratives

Updated 5 July 2026

LongStoryEval is a large-scale benchmark targeting novels over 100K tokens, addressing evaluation challenges like global coherence and reader-grounded criteria.
It systematically compares aggregation-based, incremental-updated, and summary-based evaluation methods to balance detailed analysis with computational efficiency.
The study introduces NovelCritique, an 8B evaluator that achieves nearly double the alignment with human ratings compared to larger general LLMs.

LongStoryEval is a large-scale benchmark and systematic study for automatic evaluation of book-length stories, centered on two questions: which evaluation aspects matter most to readers, and which methods remain effective when stories are far longer than typical model context windows. It comprises 600 newly published novels with an average length of 121K tokens and a maximum of 397K, associates each book with its average rating and multiple reader reviews, organizes critiques by evaluation aspects, compares aggregation-based, incremental-updated, and summary-based evaluation, and introduces the 8B evaluator NovelCritique (Yang et al., 14 Dec 2025).

1. Definition, scope, and antecedents

LongStoryEval targets book-length narratives, defined in the study as novels exceeding 100K tokens. The benchmark was introduced because prior story-evaluation datasets largely operated at much shorter scales: the survey literature summarizes representative long-form datasets such as WritingPrompts at approximately 735 tokens per story, TVStoryGen at approximately 1,868 tokens, GPT-BOOKSUM at approximately 5,363 tokens, and STORIUM at approximately 19,278 tokens, while also noting that long story evaluation remained under-explored because of hierarchical dependencies, global coherence, character arcs, worldbuilding, stylistic subjectivity, and the cost of human annotation (Yang et al., 2024).

The LongStoryEval study formulates three gaps in prior work. First, existing datasets focus on short stories in the 100–2,500-token range rather than book-scale narratives. Second, evaluation criteria had remained inconsistent and not clearly grounded in what readers actually discuss. Third, the field lacked a clear answer to how very long stories should be processed under limited context windows (Yang et al., 14 Dec 2025).

This positioning makes LongStoryEval distinct from earlier short-story evaluation settings such as HANNA and StoryER, which emphasize Likert-style assessment of shorter creative texts, and from prompt-based LLM judging studies showing that LLMs can outperform traditional metrics at system level while still struggling to provide satisfactory explanations (Chhun et al., 2024). A plausible implication is that book-length evaluation requires not only stronger evaluators, but also different input-processing regimes.

2. Benchmark construction and released data

LongStoryEval contains 600 newly published novels collected from Goodreads, published between 2024 and January 2025. To minimize contamination, the titles were verified to be absent from the pretraining corpora of the evaluated LLMs. The corpus focuses on contemporary English books from mainstream genres, and the analysis explicitly references Romance, Fantasy, Thriller, Mystery, Historical Fiction, Science Fiction, and Young Adult in its balanced aspect-distribution analysis (Yang et al., 14 Dec 2025).

The split is 450 books for training and 150 for testing. The raw review pool totals 340K reviews across the 600 books, and after bias mitigation the training set contains 176K reviews. Each book entry includes metadata such as title, genres, premise, and length; reviewer metadata when available; the book-level average star rating and rating histogram; and processed reviews reformatted into aspect-guided critiques, a brief overall assessment, and the user’s rating. Because of copyright constraints, the benchmark releases plot and character summaries instead of full texts. The anonymized test set additionally masks character names and locations to reduce future contamination risk (Yang et al., 14 Dec 2025).

Review processing uses a two-stage LLM pipeline. Raw Goodreads reviews are rewritten into aspect-guided critiques with DeepSeek-v2.5 in the first pass; if the rewritten text shares less than 40% word overlap with the original, a GPT-4o fallback is used; if overlap remains below 40%, the sample is filtered. Temperature is fixed at 0 to avoid fabrication, and manual sampling estimates 96% accuracy of these reformattings (Yang et al., 14 Dec 2025).

These design choices make LongStoryEval unusual among story benchmarks: it is built from reader-facing judgments rather than laboratory-only annotation, yet it cannot release full texts. That trade-off directly shapes the benchmark’s later emphasis on summary-based evaluation.

3. Reader-grounded criteria structure

LongStoryEval derives its rubric from reader discourse rather than from a purely top-down schema. The study automatically extracts over 1,000 user-mentioned aspects from raw reviews, then consolidates and organizes them into a hierarchical structure with eight top-level criteria (Yang et al., 14 Dec 2025).

Criterion	Core sub-aspects	Representative mappings
Plot and Structure (PLOT)	Plot development, structure, ending	pacing, conflicts, twists, coherence, closure
Characters (CHA)	Development, characterization, relationships, diversity	arc, realism, relatability, chemistry
Writing and Language (WRI)	Writing style, language, readability	prose, dialogue, grammar, clarity
World-Building and Setting (WOR)	World-building, setting	consistency, realism, cultural or historical accuracy
Themes (THE)	Exploration, clarity, depth	symbolism, motifs, messages, social commentary
Emotional Impact (EMO)	Empathy, emotional depth	resonance, range, connection to characters
Enjoyment and Engagement (ENJ)	Enjoyment, engagement	interest, excitement, immersion
Expectation Fulfillment (EXP)	Genre, premise	genre-specific expectations, premise fit

The study’s correlation analyses report that Plot and Characters are consistently the most predictive objective aspects, while Emotional Impact, Enjoyment and Engagement, and Expectation Fulfillment strongly align with ratings. Themes are helpful but secondary, whereas Writing and World-building tend to be less predictive in this 2024–2025 sample because writing and world quality are described as less variable (Yang et al., 14 Dec 2025).

This criteria structure overlaps with broader story-evaluation taxonomies catalogued in the survey literature, which also emphasize coherence, character development, empathy, interestingness, surprise, and style for long stories (Yang et al., 2024). The difference is that LongStoryEval grounds those categories directly in large-scale reader reviews, making the rubric empirically tied to audience judgment rather than solely to prior benchmark conventions.

4. Evaluation methodologies for book-length stories

LongStoryEval compares three families of methods for evaluating very long books: aggregation-based, incremental-updated, and summary-based evaluation (Yang et al., 14 Dec 2025).

Aggregation-based evaluation scores each chapter or merged short chapter separately, using book metadata, the current chapter, and a running summary of prior chapters. For an aspect $i$ , chapter-level scores $s(k,i)$ are aggregated as

$\hat{s}_i = \frac{1}{K}\sum_k s(k,i).$

This approach is described as best at capturing detail and late-appearing strengths or weaknesses.

Incremental-updated evaluation simulates a reader who revises prior judgments after each chapter. At each step, the model receives the current chapter, prior summaries, and prior critiques or scores, then produces updated critiques and updated aspect or overall scores. The study reports that this method underperforms because it combines evaluation of new text with revision of previous judgments, creating cognitive load and allowing inconsistency to accumulate (Yang et al., 14 Dec 2025).

Summary-based evaluation first constructs a compact representation of the entire book, consisting of an overall plot summary, character analysis, and selected writing excerpts. Multiple evaluators can then assess that compact representation cheaply. The paper reports that chapter-guided summaries are slightly better than a single overall summary, and that summaries produced by GPT-4o-mini are nearly as effective as more expensive summaries when paired with a strong evaluator (Yang et al., 14 Dec 2025).

The main metric for comparing evaluators is Kendall’s $\tau$ rank correlation with the human average rating:

$\tau = \frac{N_c - N_d}{n(n-1)/2}.$

A one-pass strategy that attempts to “just read the whole book” performs poorly even on a subset that fits inside 128K context, with overall $\tau \times 100$ of 5.5 for GPT-4o and 4.8 for DeepSeek-v2.5; the study attributes this to generic critiques not tied to specific plots (Yang et al., 14 Dec 2025).

Among structured methods, aggregation-based and summary-based evaluations outperform incremental-updated evaluation. For overall Kendall $\tau \times 100$ , aggregation-based evaluation gives 15.2 for GPT-4o and 15.1 for DeepSeek-v2.5, summary-based gives 13.4 for GPT-4o and 14.4 for DeepSeek-v2.5, and incremental-updated gives 10.9 for GPT-4o and 11.6 for DeepSeek-v2.5 (Yang et al., 14 Dec 2025).

The study also reports the compute trade-off on the 150-book test set with five-run averaging for GPT-4o. Summary-based evaluation uses approximately 3.94M input tokens, about 770 minutes, and about \$94; aggregation-based evaluation uses about 11.48M tokens, about 3,056 minutes, and about \$416; incremental-updated evaluation uses about 12.72M tokens, about 4,268 minutes, and about \$499 (Yang et al., 14 Dec 2025). The factual conclusion drawn in the paper is that aggregation-based evaluation excels in detail assessment, whereas summary-based evaluation offers greater efficiency.

5. NovelCritique and model-based evaluation

Building on the summary-based framework, the study introduces NovelCritique, an 8B evaluator based on Llama 3.1-8B (Yang et al., 14 Dec 2025). Its inputs are title, genres, premise, plot summary, character analysis, and writing excerpts. During training it produces aspect critiques, an overall assessment, and a normalized overall rating $S'$ . During inference it can also produce aspect scores on demand by conditioning on a specified aspect (Yang et al., 14 Dec 2025).

Training uses 450 books and 176K bias-mitigated reviews. The model is trained with standard instruction tuning under the loss

$- \log P(r_1 \ldots r_m, R, S' \mid X),$

where $s(k,i)$ 0 are aspect critiques, $s(k,i)$ 1 is the overall assessment, and $s(k,i)$ 2 is the normalized rating. User strictness is calibrated by

$s(k,i)$ 3

with normalization applied only during training (Yang et al., 14 Dec 2025).

The reported training setup is 3 epochs, learning rate $s(k,i)$ 4, batch size 32, LoRA rank $s(k,i)$ 5, $s(k,i)$ 6, and 4× A6000 GPUs for approximately 125 hours (Yang et al., 14 Dec 2025).

On the 150-book test set under summary-based evaluation, NovelCritique-8B achieves overall Kendall $s(k,i)$ 7 of 27.7, compared with 13.4 for GPT-4o, 14.4 for DeepSeek-v2.5, and 13.0 for Llama 3.1-70B. Its aspect-level scores are PLOT 27.1, CHA 27.0, WRI 24.1, WOR 18.3, THE 24.3, EMO 27.8, ENJ 21.1, and EXP 25.5 (Yang et al., 14 Dec 2025). The paper’s direct conclusion is that this domain-specialized 8B evaluator aligns nearly 2× better with human ratings than the strongest general LLM baselines used in the study.

This result is notable in the broader context of LLM-as-judge research. Earlier short-story work had already found that LLM judges can surpass classical automatic metrics at system level, but that prompting changes do not necessarily improve performance and explanations can be weakly supported (Chhun et al., 2024). LongStoryEval extends that line of work from prompt design to evaluator specialization.

6. Relation to adjacent long-story evaluation paradigms

LongStoryEval is a static benchmark for book-length narrative evaluation, but several adjacent works define complementary axes of long-story assessment.

EvolvR studies open-ended story evaluation through pairwise comparison, multi-persona score-aligned Chain-of-Thought synthesis, and multi-agent self-filtering, achieving state-of-the-art results on StoryER and HANNA and explicitly providing practical guidance for “LongStoryEval-style benchmarks.” Its proposed rubrics for long-form needs include Global Coherence, Character Consistency and Arc Progression, Plot Structure and Pacing, Style/Voice Consistency, Engagement, Empathy, Surprise, and Complexity (Wang et al., 8 Aug 2025). This suggests a close methodological fit between LongStoryEval’s reader-grounded criteria and pairwise evaluators designed for long-range dependencies.

“StoryBench” evaluates long-term memory and sequential reasoning in branching interactive fiction rather than in static novels. It uses a directed acyclic graph of 311 scene nodes and 86 choice nodes, tests both Immediate Feedback and Self Recovery settings, and measures Overall Accuracy, First-Try Accuracy, Longest Consecutive Correct Sequence, Easy/Hard Accuracy, Retry Count, Max Error, ErrorCount $s(k,i)$ 8, Runtime Cost, Token Consumption, and Success Count (Wan et al., 16 Jun 2025). The paper does not mention LongStoryEval explicitly, but it squarely targets the same thematic space of extended narrative memory and causal tracking.

“Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling” argues that rubric-based judges overlook narrative tension and introduces the 100-Endings metric, which samples 100 predicted endings after each sentence and measures mismatch with the ground-truth continuation. The paper explicitly recommends adapting mean mismatch, late-stage mismatch, inflection rate, and post-spike retention for LongStoryEval, including beat-level variants for long-form stories (Sui et al., 10 Apr 2026). In LongStoryEval terms, this adds a structural measure of suspense that is largely orthogonal to average reader ratings.

NARRA-Gym extends evaluation into interactive narrative. It measures multi-turn coherence, pacing, long-context state tracking, character simulation, empathic personalization, and artifact synthesis, using a controlled environment and an 11-dimension rubric aggregated into StoryQ and UX (Huang et al., 8 May 2026). This is not part of LongStoryEval proper, but it provides a model-in-the-loop analogue for settings where narrative quality emerges through interaction rather than through a fixed text.

Together these works indicate that LongStoryEval occupies one specific point in a larger design space: book-length, reader-grounded, static evaluation. A plausible implication is that full long-story assessment will eventually combine LongStoryEval’s review alignment with dynamic memory tests, pairwise evaluators, and explicit tension metrics.

7. Limitations, constraints, and significance

LongStoryEval inherits several constraints from its design. The released benchmark cannot include full texts because of copyright, so models are evaluated from summaries and excerpts rather than from raw novels. The dataset largely reflects mainstream English-language fiction published in 2024–2025, which limits genre and linguistic coverage. Goodreads reviews are also biased toward people with stronger opinions; the study mitigates this during training by reweighting reviews to match each book’s true rating histogram, but the ground-truth averages still reflect platform-specific preferences (Yang et al., 14 Dec 2025).

The study also notes that even incremental summarization can lose detail in very complex non-linear narratives, and that closed-model evaluators show significant variance across runs even at low temperature, making five-run averaging important but costly (Yang et al., 14 Dec 2025). These limitations align with broader survey observations that long-story evaluation remains difficult because hierarchical structure, global causality, character continuity, and reader subjectivity do not reduce cleanly to local or reference-based metrics (Yang et al., 2024).

Within those limits, LongStoryEval establishes three results that define its significance. First, it introduces the first large-scale benchmark explicitly designed for book-length story evaluation (Yang et al., 14 Dec 2025). Second, it shows that reader-grounded criteria place Plot, Characters, Emotional Impact, Enjoyment, and Expectation Fulfillment at the center of long-form evaluation rather than treating all rubric dimensions as equally informative (Yang et al., 14 Dec 2025). Third, it demonstrates that summary-based evaluation is the strongest efficiency-quality trade-off, and that a specialized 8B evaluator can outperform larger commercial models in alignment with human ratings (Yang et al., 14 Dec 2025).

In that sense, LongStoryEval turns long-story evaluation from an extension of short-story scoring into a distinct research problem: one defined by book-scale compression, reader-grounded aspect structure, and evaluator design under severe context constraints.