---
title: 'NovelCritique: LLM Evaluator for Long Stories'
url: https://www.emergentmind.com/topics/novelcritique
type: topic
---

# NovelCritique: LLM Evaluator for Long Stories

Searching arXiv for the cited works on long-story evaluation, literary quality, and critique frameworks to ground the article.
NovelCritique is an **8B-parameter open-source LLM evaluator** specialized for **book-length story evaluation**. It was introduced together with **LongStoryEval**, a benchmark of **600 newly published books** with an **average length of 121K tokens** and a **maximum of 397K**, and it is designed to produce **aspect-specific critiques**, a **global overall assessment**, and a **numerical rating on a 1.0–5.0 scale** aligned with reader ratings. Its central methodological claim is that long-story evaluation is most tractable when a book is first compressed into structured summaries and excerpts, and only then reviewed across a fixed hierarchy of reader-relevant aspects [2512.12839].

## 1. Historical and conceptual setting

NovelCritique emerged in response to a specific difficulty: automatic evaluation of long-form fiction is substantially harder than evaluating short stories or ordinary multi-paragraph texts. Three obstacles are identified. First, **data annotation constraints** make direct human evaluation of **100K–400K-token** novels expensive and cognitively demanding. Second, prior story-evaluation work uses **heterogeneous, partly overlapping criteria**, making cross-study comparison difficult. Third, even long-context LLMs remain costly and unstable when prompted with entire books, and they often produce **generic, story-agnostic critiques** [2512.12839].

The system also belongs to a longer research trajectory in computational literary evaluation. The single-blind study of anonymized Dutch novel fragments showed that judgments of literariness are shaped by both **text-intrinsic cues** and **social/contextual factors**, and that models trained on survey ratings can reproduce socially inflected judgments rather than a purified notion of intrinsic merit [2011.01624]. Corpus-scale work on English literature likewise treated literariness, creativity, and beauty as computable but multi-dimensional properties, using topic modeling, sentiment analysis, semantic complexity, forward flow, and chapter-level harmony/variety indices [2201.04356]. Against that background, NovelCritique is narrower in medium and sharper in target: it is not a general theory of literariness, but a domain-specific evaluator for **book-length stories** trained on large-scale reader critiques [2512.12839].

## 2. LongStoryEval and the criteria structure

LongStoryEval is the empirical substrate of NovelCritique. It contains **600 books**, all **newly published between 2024 and January 2025**, collected from **Goodreads**. For each book, the dataset includes metadata, rating distributions, average rating, and multiple reader reviews. Because of copyright constraints, the released version contains **book-level plot and character summaries** rather than full texts, plus review data [2512.12839].

Raw Goodreads reviews are reformatted with **DeepSeek-v2.5**, and **GPT-4o** is used if necessary, into identified aspects, aspect-specific viewpoints, a concise overall assessment, and the original **1–5** rating. Reviews whose reformatted version shares **<40% word overlap** with the original are reprocessed or dropped. The resulting resource contains **340K processed, aspect-guided reviews** [2512.12839].

By mining **>1000 user-mentioned aspects** and clustering them, the benchmark proposes **8 top-level aspects**:

| Aspect | Short definition |
|---|---|
| Plot and Structure (PLOT) | Plot development, structure, climax, ending |
| Characters (CHA) | Development, characterization, relationships, diversity |
| Writing and Language (WRI) | Style, language quality, expression |
| World-Building and Setting (WOR) | Setting construction and world detail |
| Themes (THE) | Thematic content and thematic handling |
| Emotional Impact (EMO) | Emotional force and affective response |
| Enjoyment and Engagement (ENJ) | Reading enjoyment and engagement |
| Expectation Fulfillment (EXP) | Relevance to expectations and fulfillment |

These eight aspects are not treated as equal by readers. The analysis reports that **Plot** and **Characters** are the most influential for the final rating. **Themes** are important but typically secondary. **World-building** and **Writing & Language** matter, but often show less variance. The more subjective aspects—**Emotional Impact**, **Enjoyment & Engagement**, and **Expectation Fulfillment**—also strongly correlate with overall ratings. This aspect hierarchy is the direct rationale for training NovelCritique to emit critiques structured by those same eight dimensions [2512.12839].

## 3. Long-book processing and the summary-based framework

The paper compares three strategies for evaluating long stories with LLMs: **aggregation-based**, **incremental-updated**, and **summary-based** evaluation [2512.12839].

In the **aggregation-based** framework, a model reads each chapter with metadata and prior-summary context, outputs chapter-level scores, and averages them:
$$
S_{\text{book}}^{(a)} = \frac{1}{n} \sum_{j=1}^{n} S_{j}^{(a)}.
$$
This approach uses detailed full text and is more fine-grained, but it is computationally expensive because each chapter must be processed repeatedly for stable book-level estimates.

In the **incremental-updated** framework, the evaluator mimics a reader whose judgments evolve chapter by chapter:
$$
(r^{(t)}, s^{(t)}) = f\big(c_t, \text{summary}^{(t-1)}, r^{(t-1)}, s^{(t-1)}\big).
$$
This is intuitively attractive, but the paper reports heavy instruction load and error accumulation, especially for weaker models.

NovelCritique adopts the **summary-based** framework:
$$
(r^{(a)}_i, S) = g\big(\text{Metadata}, \text{Sum}, \text{Excerpts}, \{a_1,\dots,a_m\}\big).
$$
Here the book is first compressed via **incremental summarization** into a detailed **plot summary**, **character analysis**, and representative **writing excerpts**. The summarization prompts explicitly handle **non-linear structures**, including flashbacks and alternate timelines, and maintain character records with **Profile**, **Current Experience**, and **Overall Experience** fields. The evaluator then reads those compressed representations rather than the raw full book [2512.12839].

This design makes two claims simultaneously. First, summary-based evaluation is computationally feasible under standard context windows. Second, it preserves enough narrative information to support aspect-specific critique. The reported experiments favor this trade-off: summary-based evaluation is consistently stronger than incremental-updated evaluation and close to aggregation-based evaluation while being far cheaper [2512.12839].

## 4. Model architecture, instruction tuning, and score normalization

NovelCritique is built by instruction tuning **Llama 3.1-8B-Instruct** with **LoRA** using **\(r = 64\)** and **\(\alpha = 16\)** [2512.12839]. Its training inputs consist of:

- **Title, genres, premise**
- **Incremental plot summary**
- **Character analysis**
- **Writing excerpts**
- A **criteria list** specifying any subset of the eight aspects

Its output format is structured as aspect-specific critiques, an overall assessment, and a scalar score. The intended inference schema follows the same design, allowing targeted evaluation of only selected aspects when desired [2512.12839].

Training uses **450** of the **600** books in LongStoryEval. After review-bias mitigation, the effective training set contains **176K reviews**. The bias-mitigation step addresses the fact that moderate ratings are less likely to generate reviews than rating extremes. The paper reports approximate review ratios of **5★: 22%**, **4★: 19%**, **3★: 17%**, **2★: 22%**, and **1★: 31%**, and it subsamples reviews per book to better match the underlying rating distribution [2512.12839].

A second normalization addresses reviewer harshness or leniency. During training, the review score is transformed as:
$$
S' = \frac{S - \mu_u}{\sigma_u} \times \sigma_{\text{plat}} + \mu_{\text{plat}},
$$
where \(S\) is the original rating, \(\mu_u\) and \(\sigma_u\) are the reviewer’s personal mean and standard deviation, and \(\mu_{\text{plat}}\) and \(\sigma_{\text{plat}}\) are platform-wide statistics. The model is then trained with standard **supervised instruction tuning** under **cross-entropy loss**. The reported setup uses **3 epochs**, **learning rate \(1\times10^{-5}\)**, **batch size 32**, and **four A6000 GPUs** for about **125 hours** [2512.12839].

The resulting system is therefore not merely a general LLM prompted to act as a reviewer. It is a domain-specialized evaluator aligned to reader-review distributions, aspect structure, and long-book summarization conventions.

## 5. Empirical performance and methodological comparisons

Evaluation uses **system-level Kendall’s tau** between model-predicted scores and average Goodreads ratings, reported as **\(\tau \times 100\)** for readability [2512.12839]. The headline result is that NovelCritique substantially outperforms strong open and closed baselines in the **summary-based** setting.

| Aspect | GPT-4o (summary-based) | NovelCritique-8B |
|---|---:|---:|
| PLOT | 15.3 | 27.1 |
| CHA | 17.8 | 27.0 |
| WRI | 4.5 | 24.1 |
| WOR | 5.0 | 18.3 |
| THE | 7.2 | 24.3 |
| EMO | 12.6 | 27.8 |
| ENJ | 11.8 | 21.1 |
| EXP | 14.0 | 25.5 |
| Overall | 13.4 | 27.7 |

The same table shows DeepSeek-v2.5 at **14.4** overall, GPT-4o-mini at **9.7**, Llama 3.1-70B-Instruct at **13.0**, Llama 3.1-8B-Instruct at **12.4**, and Mixtral 8×7B-Instruct at **8.3**. NovelCritique’s gains are especially large on **Emotional Impact**, **Plot**, **Characters**, and **Expectation Fulfillment**, indicating that the model aligns well not only with relatively objective narrative dimensions but also with reader-experience dimensions [2512.12839].

The method comparison also matters. For GPT-4o, the reported overall correlations are:

- **One-pass**: **5.5**
- **Incremental-updated**: **10.9**
- **Summary-based**: **13.4**
- **Aggregation-based**: **15.2**

For Llama 3.1-70B, **aggregation-based** reaches **13.8**, **summary-based** **13.0**, and **incremental-updated** **9.9**. The conclusion is consistent across models: **aggregation-based** and **summary-based** evaluation outperform **incremental-updated**, while summary-based evaluation offers the most favorable cost-accuracy trade-off [2512.12839].

Ablation analysis further shows that performance drops when removing **review organization**, **review bias mitigation**, or **rating normalization**, which indicates that the model’s alignment is not produced by base-model scale alone. It depends on the structure of the review data and the normalization of user preferences [2512.12839].

## 6. Relation to adjacent research programs

NovelCritique sits alongside several adjacent but methodologically distinct lines of work. The blind literary taste-test study on Dutch novels demonstrated that fragment-based literariness judgments remain positively correlated with survey-based judgments, while genre and gender penalties largely disappear when texts are anonymized. That result frames literary assessment as a joint product of textual cues and social prestige structures rather than a purely intrinsic property [2011.01624]. NovelCritique differs by training directly on large-scale reader critiques of full book experiences rather than on blind fragment ratings.

The GLEC-based work by Jacobs and Kinder proposed a multi-dimensional computational critique framework using **topic modeling**, **SentiArt**, **intra-textual variance**, **stepwise distance**, **forward flow**, and **harmony/variety** measures. It showed predictive accuracies in the range of **.75 to .97** for text classification and authorship recognition, and it treated literariness, creativity, and beauty as partially separable semantic-complexity phenomena [2201.04356]. NovelCritique is less theory-laden in its feature design, but similar in its insistence that literary evaluation should be multi-dimensional rather than reduced to a single score.

More recent work on **SAGE** pushes further toward interpretive depth by decomposing literary quality into ontology-grounded **cultural**, **emotional-psychological**, and **existential-philosophical** layers. Across **600 evaluations**, SAGE reports **98.8% score convergence**, **greater than 94% inter-rater agreement**, and a consistent **Canonical > Pulp > LLM** hierarchy, with especially large gaps on cultural critique and philosophical depth [2605.07102]. This suggests that NovelCritique’s eight-aspect structure captures a reader-centered evaluation space, but not the full ontology of interpretive literary criticism.

Work on **LLM Review** contributes a different lesson: in creative writing, **Blind Peer Review** among LLM agents can improve stories while preserving divergence, and a combined evaluation framework using **LLM-as-a-judge**, **human annotation**, and **rule-based novelty metrics** can detect gains in both quality and novelty [2601.08003]. That line is closer to revision-oriented writing assistance, whereas NovelCritique is an evaluator; still, both assume that long-form fiction requires structured, aspect-sensitive critique rather than generic summarization.

Finally, the study of textual creativity beyond n-gram novelty demonstrates that novelty alone is a poor surrogate for creative merit: about **91% of top-quartile expressions by n-gram novelty are not judged as creative** by expert writers, and higher n-gram novelty in open-source LLMs correlates with lower pragmaticality [2509.22641]. This reinforces NovelCritique’s choice to evaluate books across multiple dimensions, including **Emotional Impact**, **Enjoyment & Engagement**, and **Expectation Fulfillment**, instead of relying on novelty-like lexical signals alone.

## 7. Limitations and prospective development

The authors state that **generative evaluation remains inconsistent**, even with **temperature = 0** and **5-run averaging** for closed models [2512.12839]. The benchmark is also **English-centric**, grounded in Goodreads reviews, and concentrated in popular genres such as Romance, Fantasy, Thriller, Mystery, Historical Fiction, Science Fiction, and Young Adult. Some niche genres and non-English literary traditions are underrepresented. The model therefore inherits not only the strengths but also the cultural and demographic biases of its source platform.

A second limitation is the dependence on **LLM-generated summaries**. NovelCritique evaluates books through **incremental GPT-4o summaries**, character analyses, and excerpts rather than full texts. The paper notes that such summaries may omit or distort important details, even if experiments show that cheaper **GPT-4o-mini summaries** cause little degradation in downstream performance [2512.12839].

A third limitation is that evaluation is **general rather than personalized**. Reviewer metadata exists, but the model is trained to align with aggregate reader judgments, not with identifiable taste communities or individual preference profiles. The authors explicitly note **personalized evaluation** as a future direction, alongside alternatives to scalar scoring such as **pairwise comparison frameworks** and more faithful use of full texts [2512.12839].

Taken together, these constraints define the current scope of NovelCritique. It is best understood as a specialized reader-aligned evaluator for long fiction, built on large-scale aspect-guided review data and a summary-based long-context pipeline. It does not replace human criticism, and it does not claim to model intrinsic literary merit in a normatively final sense. Its importance lies in showing that **domain-specific LLM evaluators trained on structured reader discourse can achieve substantially better alignment with human long-story judgments than general-purpose frontier models**, while also exposing where future systems may need richer interpretive ontologies, better personalization, and more faithful access to full narrative structure [2512.12839].

Source: https://www.emergentmind.com/topics/novelcritique