NarraBench: A Framework for Narrative Benchmarking
- NarraBench is a theory-informed framework that defines narrative understanding through a taxonomy of 50 tasks spanning story content, narration, discourse, and situatedness.
- It critiques current benchmarks for being overly deterministic and narrowly focused on factual, single-answer tasks, urging a shift toward multi-perspective evaluation.
- The framework empirically aligns 34 open benchmarks to its taxonomy and reveals that only about 27% of the narrative task space is adequately covered.
Searching arXiv for NarraBench and closely related narrative-benchmark papers. NarraBench is a theory-informed framework for evaluating narrative understanding in LLMs. It is presented as both a taxonomy of narrative-understanding tasks, derived from narratology, and a survey and organizational framework for existing benchmarks. Its central claim is that current evaluations test narrative competence in fragmented, theoretically inconsistent, and often overly narrow ways, with particular overemphasis on factual, single-answer, story-level tasks and weak coverage of events, style, perspective, revelation, and socially situated interpretation. The paper estimates that only about 27% of the task space in the NarraBench taxonomy is well covered by existing benchmarks (Hamilton et al., 10 Oct 2025).
1. Rationale and scope
NarraBench was introduced to address what it describes as a structural inadequacy in narrative benchmarking. Narrative understanding is treated as central to human communication and already implicit in many NLP tasks, yet existing benchmarks are said to be under-theorized in evaluation design, mostly deterministic in labeling, overly focused on “story content,” and disconnected from a shared framework tying narrative theory to concrete task types and evaluation modes. In this formulation, a benchmark that asks only who did what in a story captures only a narrow slice of narrative intelligence (Hamilton et al., 10 Oct 2025).
The framework therefore should not be understood as a single executable environment, a fixed dataset, or a leaderboard over one task. Its unit of analysis is broader: it provides a benchmarking ontology, a methodology for aligning existing resources to that ontology, and an empirical diagnosis of where benchmark coverage is missing or mis-specified. A common misconception is to treat NarraBench as merely another reading-comprehension benchmark; the paper instead presents it as infrastructure for curating, auditing, and building benchmarks across the full space of narrative understanding (Hamilton et al., 10 Oct 2025).
2. Conceptual model of narrative
The framework grounds its definition of narrative communication in a schema drawn from Piper et al. A narrative occurs when all of the following are present: A Someone, B tells, C someone, D somewhere, E for some reason, that, F someone, G did something(s), H [to/with someone], I somewhere, J at some time, K for some reason. NarraBench uses this schema to distinguish the frame of telling—A–E—from the storyworld frame—F–K. The former is glossed as heterodiegetic elements and the latter as diegetic elements (Hamilton et al., 10 Oct 2025).
On top of this minimal schema, the paper emphasizes that narratives centrally involve change, conflict, temporality, and agent-centered events. Narrative understanding is therefore defined more broadly than factual recovery. It includes storyworld content, the way the story is told, who speaks and from what vantage, why the narrative is communicated, and where interpretation is non-unique and socially situated (Hamilton et al., 10 Oct 2025).
The taxonomy is organized around four high-level dimensions, combining Genette’s story/discourse/narration triangle with Herman’s situatedness:
| Dimension | Focus | Primary features |
|---|---|---|
| Story | Storyworld content | Agents, Social Networks, Events, Plot, Structure, Setting |
| Narration | Who speaks and how | Perspective, Style |
| Discourse | Organization and disclosure | Time, Revelation |
| Situatedness | Real-world communicative context | Paratext, Motivation |
This synthesis is one of the framework’s defining features. It makes explicit that narrative understanding is not reducible to plot extraction, because narration, discourse organization, and social context are treated as first-class benchmark targets rather than peripheral metadata (Hamilton et al., 10 Oct 2025).
3. Taxonomy of tasks and evaluative attributes
The central formal artifact in NarraBench is a taxonomy of 50 narrative-understanding tasks, organized through the four high-level dimensions and twelve primary features. Within Story, the framework includes Agents, Social Networks, Events, Plot, Structure, and Setting. Within Narration, it includes Perspective and Style. Within Discourse, it includes Time and Revelation. Within Situatedness, it includes Paratext and Motivation (Hamilton et al., 10 Oct 2025).
The feature inventory is deliberately granular. Agents cover names, roles, attributes, emotions, and motivation. Social Networks cover interaction, connections, and relationship. Events cover event identification, schema, and causality. Plot includes topic, plot, plotline, moral, obstacle, conflict, and archetype. Structure contains plot arc. Setting distinguishes setting and location. Perspective separates point of view, focalization, and dialogue attribution. Style includes allusion, figurative language, imageability, complexity, and evaluative discourse. Time covers duration and order. Revelation includes suspense, curiosity, and surprise. Paratext includes genre, author, date, medium, and platform. Motivation covers authorial intent (Hamilton et al., 10 Oct 2025).
NarraBench does not classify tasks only by content area. Each task is also specified along three orthogonal evaluative criteria: scale, mode, and variance. Scale distinguishes local, meso, and global phenomena. Mode distinguishes discrete, progressive, and holistic outputs. Variance distinguishes deterministic, consensus, and perspectival tasks. This last axis is especially important in the framework’s argument: some narrative questions have one correct answer, some admit multiple answers with a stable central tendency, and some are constitutively interpretive with no uniquely correct response. NarraBench argues that benchmark design should respect this variance structure rather than coercing all tasks into deterministic labeling (Hamilton et al., 10 Oct 2025).
The framework’s treatment of variance marks a significant shift from standard NLP benchmark design. Character motivation, moral, conflict, authorial intent, suspense, imageability, and allusion are not treated as annotation problems that merely need stricter gold labels; they are treated as tasks whose ontology may require consensus-based or perspectival evaluation. In the paper’s view, misalignment between task ontology and scoring protocol is itself a benchmark-design error (Hamilton et al., 10 Oct 2025).
4. Survey methodology and benchmark alignment
NarraBench accompanies its taxonomy with a survey of 78 candidate benchmarks from the last twelve years that might be relevant to narrative understanding. A benchmark qualified for inclusion only if it yielded results useful for adjudicating model performance on one or more of the twelve narrative features, had code and data in a repository, provided convenience functions or tooling for testing arbitrary LLMs via standard APIs, and served as a comparative metric across models rather than a one-off system-specific evaluation (Hamilton et al., 10 Oct 2025).
From these 78 candidates, 39 lacked available data and 39 had usable open repositories and data. NarraBench then aligns each benchmark to taxonomy cells using an edit-distance-style fit measure over three properties: scale, mode, and variance. Quality labels are assigned as good for distance 0, decent for distance 1, poor for distance 2, and bad for distance 3. The framework retains benchmarks of poor-or-better quality, leaving 34 usable benchmarks for coverage analysis (Hamilton et al., 10 Oct 2025).
| Quality | Distance | Benchmark count |
|---|---|---|
| Good | 0 | 10 |
| Decent | 1 | 14 |
| Poor | 2 | 10 |
| Bad (ignored) | 3 | 5 |
| Missing data | - | 39 |
The methodological point is that benchmark coverage is not binary. A benchmark may nominally target the right narrative phenomenon while still mismatching its proper evaluative form. A task that is intrinsically perspectival but benchmarked deterministically, or progressive but collapsed into a single holistic answer, counts as misaligned rather than fully representative. This turns NarraBench into a framework for evaluating benchmarks themselves, not only LLMs (Hamilton et al., 10 Oct 2025).
The survey also documents current methodological tendencies. Among the 39 surveyed open benchmarks, 27 use classification or multi-label prediction and 12 evaluate open-ended generation. Only 4 of the 39 shortlisted benchmarks are multilingual, and about 5% of surveyed benchmarks are multimodal or non-textual. These counts support the paper’s broader claim that the benchmark ecosystem is both methodologically narrow and linguistically concentrated (Hamilton et al., 10 Oct 2025).
5. Coverage findings and diagnosed blind spots
The framework’s headline empirical finding is that existing benchmarks cover only approximately 27% of the NarraBench taxonomy. This is not a ratio of benchmark count to task count. Rather, the survey finds that the 34 usable benchmarks cluster on a limited subset of the 50 taxonomy cells, leaving large parts of the space either unbenchmarked or only poorly aligned (Hamilton et al., 10 Oct 2025).
Coverage is highly imbalanced across the four major dimensions. NarraBench reports 19 benchmarks for story-specific features, but only 2 for narration, 5 for discourse, and 5 for situatedness. Current evaluation therefore concentrates on character facts, plot content, and other storyworld retrieval tasks, while giving comparatively little attention to voice, viewpoint, discourse organization, and communicative context (Hamilton et al., 10 Oct 2025).
Several omissions are especially sharp. The paper reports no benchmarks meeting its criteria for event, schema, or causality, despite the centrality of events to narrative structure. It also finds no benchmarks for style features such as allusion, figurative language, imageability, complexity, or evaluative discourse. Perspective is severely under-evaluated: dialogue attribution has limited coverage, while point of view and focalization are nearly absent. Revelation is similarly sparse: there is one benchmark for suspense, but none for curiosity or surprise (Hamilton et al., 10 Oct 2025).
NarraBench also criticizes the prevailing assumption that narrative tasks should be benchmarked deterministically. The survey reports that all but two surveyed benchmarks construct ground truth deterministically. For the framework, this is not just a stylistic concern about annotation design. It implies that many current evaluations confound narrative understanding with conformity to annotation compression. A benchmark that scores a single moral, single emotion, or single interpretation as correct may systematically undercount valid alternative readings (Hamilton et al., 10 Oct 2025).
These blind spots matter because they mark the boundary between shallow story QA and broader narrative competence. Without event and causal structure, a model may recover surface facts without grasping plot logic. Without style, it may miss irony, symbolism, and evaluative framing. Without perspective, it may flatten distinct consciousnesses and misattribute knowledge or perception. Without revelation, it may fail to model suspense, mystery, or strategic information withholding. NarraBench’s diagnosis is therefore that current benchmarks often measure understanding of stories as information containers rather than narratives as narratives (Hamilton et al., 10 Oct 2025).
6. Subsequent relevance, adjacent benchmarks, and open problems
NarraBench’s taxonomy has become a useful reference point for later work that operationalizes narrative evaluation more concretely. "StoryScope" (Russell et al., 3 Apr 2026) adopts ten NarraBench dimensions—Agent, Social Network, Event, Plot, Structure, Setting, Time, Revelation, Perspective, and Style—while excluding Paratext and Motivation, and then induces an interpretable feature space with 304 extracted features per story. In that setting, narrative features alone achieve 93.2% macro-F1 for human-vs.-AI detection and 68.4% macro-F1 for six-way authorship attribution, indicating that discourse-level narrative structure can be measured at scale rather than treated only as a theoretical desideratum (Russell et al., 3 Apr 2026).
Other benchmark efforts occupy adjacent parts of the space that NarraBench identifies as underdeveloped. "NARRA-Gym" (Huang et al., 8 May 2026) is an executable evaluation environment for interactive narrative agents, emphasizing multi-turn story generation, long-horizon coherence, user adaptation, pacing, character shaping, and UX rather than static prompt-response evaluation. "VinaBench" (Gao et al., 26 Mar 2025) addresses visual narrative generation, focusing on commonsense grounding, discourse constraints, faithfulness to textual narrative, and consistency across image sequences. "KnowMe-Bench" (Wu et al., 8 Jan 2026) uses long-form autobiographical narratives to evaluate factual recall, subjective state attribution, and principle-level reasoning for person understanding, thereby extending narrative benchmarking toward evidence-grounded companion modeling rather than generic story comprehension. "AI as a Tool for Simulation-Based Experiments in Literary Studies" (Wilkens, 1 Jun 2026) argues that literary generation should be assessed not only for aesthetic output but also for controllability, condition sensitivity, and distributional realism, while explicitly noting that coherence, sustained character development, and interpretive richness remain insufficiently measured (Wilkens, 1 Jun 2026).
These neighboring projects do not replace NarraBench’s role. Rather, they illustrate the division of labor that the framework itself implies: NarraBench maps the task space and its evaluative ontology, while later works instantiate narrower regions of that space with concrete datasets, feature pipelines, or executable environments. A plausible implication is that NarraBench is most valuable when used as a reference layer for deciding which kinds of narrative competence a benchmark actually targets, and whether its scoring protocol matches the narrative phenomenon under evaluation.
The paper is also explicit about its own limitations. It is rooted in a classical narratology lineage and does not fully incorporate cognitive narratology, rhetorical theory, or other postclassical approaches centered on affect, ideology, or embodiment. Because the survey only analyzes public and open benchmarks, it is likely biased toward English, Western corpora, and text-only tasks. Its modular decomposition of narrative into separable dimensions is analytically useful but may understate how strongly style, revelation, perspective, and social context interact in actual interpretation. The reported 27% figure is a coverage estimate over the proposed taxonomy, not a validated measure of total narrative competence. NarraBench also inherits unresolved problems common to generative evaluation, including instability of LLM-as-a-judge, reproducibility challenges, and the difficulty of encoding human interpretive diversity in benchmark gold standards (Hamilton et al., 10 Oct 2025).
In this sense, NarraBench is best understood not as a finished benchmark suite but as a comprehensive framework for narrative benchmarking. Its lasting contribution is to formalize the claim that narrative understanding is multidimensional, theoretically structured, and frequently non-deterministic. By doing so, it reorients benchmark design away from narrow factual QA and toward a fuller accounting of events, style, perspective, revelation, situatedness, and interpretive plurality (Hamilton et al., 10 Oct 2025).