NovelCritique: LLM Evaluator for Long Stories

• The paper demonstrates a novel approach by compressing long novels into structured summaries and excerpts to produce aspect-specific critiques and numerical ratings on a 1.0–5.0 scale.
• It leverages the LongStoryEval benchmark of 600 Goodreads-based books to assess eight key literary dimensions, including Plot, Characters, and Emotional Impact, achieving superior alignment with reader reviews.
• Method comparisons reveal that the summary-based evaluation strategy offers a competitive cost-accuracy trade-off against aggregation-based and incremental-updated methodologies.

Searching arXiv for the cited works on long-story evaluation, literary quality, and critique frameworks to ground the article. NovelCritique is an 8B-parameter open-source LLM evaluator specialized for book-length story evaluation. It was introduced together with LongStoryEval, a benchmark of 600 newly published books with an average length of 121K tokens and a maximum of 397K, and it is designed to produce aspect-specific critiques, a global overall assessment, and a numerical rating on a 1.0–5.0 scale aligned with reader ratings. Its central methodological claim is that long-story evaluation is most tractable when a book is first compressed into structured summaries and excerpts, and only then reviewed across a fixed hierarchy of reader-relevant aspects (Yang et al., 14 Dec 2025).

1. Historical and conceptual setting

NovelCritique emerged in response to a specific difficulty: automatic evaluation of long-form fiction is substantially harder than evaluating short stories or ordinary multi-paragraph texts. Three obstacles are identified. First, data annotation constraints make direct human evaluation of 100K–400K-token novels expensive and cognitively demanding. Second, prior story-evaluation work uses heterogeneous, partly overlapping criteria, making cross-study comparison difficult. Third, even long-context LLMs remain costly and unstable when prompted with entire books, and they often produce generic, story-agnostic critiques (Yang et al., 14 Dec 2025).

The system also belongs to a longer research trajectory in computational literary evaluation. The single-blind study of anonymized Dutch novel fragments showed that judgments of literariness are shaped by both text-intrinsic cues and social/contextual factors, and that models trained on survey ratings can reproduce socially inflected judgments rather than a purified notion of intrinsic merit (Cranenburgh et al., 2020). Corpus-scale work on English literature likewise treated literariness, creativity, and beauty as computable but multi-dimensional properties, using topic modeling, sentiment analysis, semantic complexity, forward flow, and chapter-level harmony/variety indices (Jacobs et al., 2022). Against that background, NovelCritique is narrower in medium and sharper in target: it is not a general theory of literariness, but a domain-specific evaluator for book-length stories trained on large-scale reader critiques (Yang et al., 14 Dec 2025).

2. LongStoryEval and the criteria structure

LongStoryEval is the empirical substrate of NovelCritique. It contains 600 books, all newly published between 2024 and January 2025, collected from Goodreads. For each book, the dataset includes metadata, rating distributions, average rating, and multiple reader reviews. Because of copyright constraints, the released version contains book-level plot and character summaries rather than full texts, plus review data (Yang et al., 14 Dec 2025).

Raw Goodreads reviews are reformatted with DeepSeek-v2.5, and GPT-4o is used if necessary, into identified aspects, aspect-specific viewpoints, a concise overall assessment, and the original 1–5 rating. Reviews whose reformatted version shares <40% word overlap with the original are reprocessed or dropped. The resulting resource contains 340K processed, aspect-guided reviews (Yang et al., 14 Dec 2025).

By mining >1000 user-mentioned aspects and clustering them, the benchmark proposes 8 top-level aspects:

Aspect	Short definition
Plot and Structure (PLOT)	Plot development, structure, climax, ending
Characters (CHA)	Development, characterization, relationships, diversity
Writing and Language (WRI)	Style, language quality, expression
World-Building and Setting (WOR)	Setting construction and world detail
Themes (THE)	Thematic content and thematic handling
Emotional Impact (EMO)	Emotional force and affective response
Enjoyment and Engagement (ENJ)	Reading enjoyment and engagement
Expectation Fulfillment (EXP)	Relevance to expectations and fulfillment

These eight aspects are not treated as equal by readers. The analysis reports that Plot and Characters are the most influential for the final rating. Themes are important but typically secondary. World-building and Writing & Language matter, but often show less variance. The more subjective aspects—Emotional Impact, Enjoyment & Engagement, and Expectation Fulfillment—also strongly correlate with overall ratings. This aspect hierarchy is the direct rationale for training NovelCritique to emit critiques structured by those same eight dimensions (Yang et al., 14 Dec 2025).

3. Long-book processing and the summary-based framework

The paper compares three strategies for evaluating long stories with LLMs: aggregation-based, incremental-updated, and summary-based evaluation (Yang et al., 14 Dec 2025).

In the aggregation-based framework, a model reads each chapter with metadata and prior-summary context, outputs chapter-level scores, and averages them:

$$S_{\text{book}}^{(a)} = \frac{1}{n} \sum_{j=1}^{n} S_{j}^{(a)}.$$

This approach uses detailed full text and is more fine-grained, but it is computationally expensive because each chapter must be processed repeatedly for stable book-level estimates.

In the incremental-updated framework, the evaluator mimics a reader whose judgments evolve chapter by chapter:

$$(r^{(t)}, s^{(t)}) = f\big(c_t, \text{summary}^{(t-1)}, r^{(t-1)}, s^{(t-1)}\big).$$

This is intuitively attractive, but the paper reports heavy instruction load and error accumulation, especially for weaker models.

NovelCritique adopts the summary-based framework:

$$(r^{(a)}_i, S) = g\big(\text{Metadata}, \text{Sum}, \text{Excerpts}, \{a_1,\dots,a_m\}\big).$$

Here the book is first compressed via incremental summarization into a detailed plot summary, character analysis, and representative writing excerpts. The summarization prompts explicitly handle non-linear structures, including flashbacks and alternate timelines, and maintain character records with Profile, Current Experience, and Overall Experience fields. The evaluator then reads those compressed representations rather than the raw full book (Yang et al., 14 Dec 2025).

This design makes two claims simultaneously. First, summary-based evaluation is computationally feasible under standard context windows. Second, it preserves enough narrative information to support aspect-specific critique. The reported experiments favor this trade-off: summary-based evaluation is consistently stronger than incremental-updated evaluation and close to aggregation-based evaluation while being far cheaper (Yang et al., 14 Dec 2025).

4. Model architecture, instruction tuning, and score normalization

NovelCritique is built by instruction tuning Llama 3.1-8B-Instruct with LoRA using $r = 64$ and $\alpha = 16$ (Yang et al., 14 Dec 2025). Its training inputs consist of:

• Title, genres, premise
• Incremental plot summary
• Character analysis
• Writing excerpts
• A criteria list specifying any subset of the eight aspects

Its output format is structured as aspect-specific critiques, an overall assessment, and a scalar score. The intended inference schema follows the same design, allowing targeted evaluation of only selected aspects when desired (Yang et al., 14 Dec 2025).

Training uses 450 of the 600 books in LongStoryEval. After review-bias mitigation, the effective training set contains 176K reviews. The bias-mitigation step addresses the fact that moderate ratings are less likely to generate reviews than rating extremes. The paper reports approximate review ratios of 5★: 22%, 4★: 19%, 3★: 17%, 2★: 22%, and 1★: 31%, and it subsamples reviews per book to better match the underlying rating distribution (Yang et al., 14 Dec 2025).

A second normalization addresses reviewer harshness or leniency. During training, the review score is transformed as:

$$S' = \frac{S - \mu_u}{\sigma_u} \times \sigma_{\text{plat}} + \mu_{\text{plat}},$$

where $S$ is the original rating, $\mu_u$ and $\sigma_u$ are the reviewer’s personal mean and standard deviation, and $\mu_{\text{plat}}$ and $$(r^{(t)}, s^{(t)}) = f\big(c_t, \text{summary}^{(t-1)}, r^{(t-1)}, s^{(t-1)}\big).$$0 are platform-wide statistics. The model is then trained with standard supervised instruction tuning under cross-entropy loss. The reported setup uses 3 epochs, learning rate $$(r^{(t)}, s^{(t)}) = f\big(c_t, \text{summary}^{(t-1)}, r^{(t-1)}, s^{(t-1)}\big).$$1, batch size 32, and four A6000 GPUs for about 125 hours (Yang et al., 14 Dec 2025).

The resulting system is therefore not merely a general LLM prompted to act as a reviewer. It is a domain-specialized evaluator aligned to reader-review distributions, aspect structure, and long-book summarization conventions.

5. Empirical performance and methodological comparisons

Evaluation uses system-level Kendall’s tau between model-predicted scores and average Goodreads ratings, reported as $$(r^{(t)}, s^{(t)}) = f\big(c_t, \text{summary}^{(t-1)}, r^{(t-1)}, s^{(t-1)}\big).$$2 for readability (Yang et al., 14 Dec 2025). The headline result is that NovelCritique substantially outperforms strong open and closed baselines in the summary-based setting.

Aspect	GPT-4o (summary-based)	NovelCritique-8B
PLOT	15.3	27.1
CHA	17.8	27.0
WRI	4.5	24.1
WOR	5.0	18.3
THE	7.2	24.3
EMO	12.6	27.8
ENJ	11.8	21.1
EXP	14.0	25.5
Overall	13.4	27.7

The same table shows DeepSeek-v2.5 at 14.4 overall, GPT-4o-mini at 9.7, Llama 3.1-70B-Instruct at 13.0, Llama 3.1-8B-Instruct at 12.4, and Mixtral 8×7B-Instruct at 8.3. NovelCritique’s gains are especially large on Emotional Impact, Plot, Characters, and Expectation Fulfillment, indicating that the model aligns well not only with relatively objective narrative dimensions but also with reader-experience dimensions (Yang et al., 14 Dec 2025).

The method comparison also matters. For GPT-4o, the reported overall correlations are:

• One-pass: 5.5
• Incremental-updated: 10.9
• Summary-based: 13.4
• Aggregation-based: 15.2

For Llama 3.1-70B, aggregation-based reaches 13.8, summary-based 13.0, and incremental-updated 9.9. The conclusion is consistent across models: aggregation-based and summary-based evaluation outperform incremental-updated, while summary-based evaluation offers the most favorable cost-accuracy trade-off (Yang et al., 14 Dec 2025).

Ablation analysis further shows that performance drops when removing review organization, review bias mitigation, or rating normalization, which indicates that the model’s alignment is not produced by base-model scale alone. It depends on the structure of the review data and the normalization of user preferences (Yang et al., 14 Dec 2025).

6. Relation to adjacent research programs

NovelCritique sits alongside several adjacent but methodologically distinct lines of work. The blind literary taste-test study on Dutch novels demonstrated that fragment-based literariness judgments remain positively correlated with survey-based judgments, while genre and gender penalties largely disappear when texts are anonymized. That result frames literary assessment as a joint product of textual cues and social prestige structures rather than a purely intrinsic property (Cranenburgh et al., 2020). NovelCritique differs by training directly on large-scale reader critiques of full book experiences rather than on blind fragment ratings.

The GLEC-based work by Jacobs and Kinder proposed a multi-dimensional computational critique framework using topic modeling, SentiArt, intra-textual variance, stepwise distance, forward flow, and harmony/variety measures. It showed predictive accuracies in the range of .75 to .97 for text classification and authorship recognition, and it treated literariness, creativity, and beauty as partially separable semantic-complexity phenomena (Jacobs et al., 2022). NovelCritique is less theory-laden in its feature design, but similar in its insistence that literary evaluation should be multi-dimensional rather than reduced to a single score.

More recent work on SAGE pushes further toward interpretive depth by decomposing literary quality into ontology-grounded cultural, emotional-psychological, and existential-philosophical layers. Across 600 evaluations, SAGE reports 98.8% score convergence, greater than 94% inter-rater agreement, and a consistent Canonical > Pulp > LLM hierarchy, with especially large gaps on cultural critique and philosophical depth (Wang et al., 8 May 2026). This suggests that NovelCritique’s eight-aspect structure captures a reader-centered evaluation space, but not the full ontology of interpretive literary criticism.

Work on LLM Review contributes a different lesson: in creative writing, Blind Peer Review among LLM agents can improve stories while preserving divergence, and a combined evaluation framework using LLM-as-a-judge, human annotation, and rule-based novelty metrics can detect gains in both quality and novelty (Li et al., 12 Jan 2026). That line is closer to revision-oriented writing assistance, whereas NovelCritique is an evaluator; still, both assume that long-form fiction requires structured, aspect-sensitive critique rather than generic summarization.

Finally, the study of textual creativity beyond n-gram novelty demonstrates that novelty alone is a poor surrogate for creative merit: about 91% of top-quartile expressions by n-gram novelty are not judged as creative by expert writers, and higher n-gram novelty in open-source LLMs correlates with lower pragmaticality (Saakyan et al., 26 Sep 2025). This reinforces NovelCritique’s choice to evaluate books across multiple dimensions, including Emotional Impact, Enjoyment & Engagement, and Expectation Fulfillment, instead of relying on novelty-like lexical signals alone.

7. Limitations and prospective development

The authors state that generative evaluation remains inconsistent, even with temperature = 0 and 5-run averaging for closed models (Yang et al., 14 Dec 2025). The benchmark is also English-centric, grounded in Goodreads reviews, and concentrated in popular genres such as Romance, Fantasy, Thriller, Mystery, Historical Fiction, Science Fiction, and Young Adult. Some niche genres and non-English literary traditions are underrepresented. The model therefore inherits not only the strengths but also the cultural and demographic biases of its source platform.

A second limitation is the dependence on LLM-generated summaries. NovelCritique evaluates books through incremental GPT-4o summaries, character analyses, and excerpts rather than full texts. The paper notes that such summaries may omit or distort important details, even if experiments show that cheaper GPT-4o-mini summaries cause little degradation in downstream performance (Yang et al., 14 Dec 2025).

A third limitation is that evaluation is general rather than personalized. Reviewer metadata exists, but the model is trained to align with aggregate reader judgments, not with identifiable taste communities or individual preference profiles. The authors explicitly note personalized evaluation as a future direction, alongside alternatives to scalar scoring such as pairwise comparison frameworks and more faithful use of full texts (Yang et al., 14 Dec 2025).

Taken together, these constraints define the current scope of NovelCritique. It is best understood as a specialized reader-aligned evaluator for long fiction, built on large-scale aspect-guided review data and a summary-based long-context pipeline. It does not replace human criticism, and it does not claim to model intrinsic literary merit in a normatively final sense. Its importance lies in showing that domain-specific LLM evaluators trained on structured reader discourse can achieve substantially better alignment with human long-story judgments than general-purpose frontier models, while also exposing where future systems may need richer interpretive ontologies, better personalization, and more faithful access to full narrative structure (Yang et al., 14 Dec 2025).