---
title: Story Reasoning Dataset
url: https://www.emergentmind.com/topics/story-reasoning-dataset
type: topic
---

# Story Reasoning Dataset

Searching arXiv for recent and foundational papers on story reasoning datasets and related benchmarks.
I’m checking arXiv entries relevant to story reasoning, narrative QA, and story generation benchmarks to ground the article in current literature.
A story reasoning dataset is a narrative benchmark in which success depends on more than surface fluency or local lexical matching. Across the literature, such datasets operationalize reasoning through tasks such as interactive disambiguation, counterfactual rewriting, long-context continuation under evolving narrative state, multimodal alignment between stories and images or video, deep video understanding over storylines, and reader-response inference conditioned on social context [1708.01776][2010.01717][2406.10221][2512.15925]. Taken together, these resources suggest that story reasoning is not a single benchmark format but a family of task designs centered on plot progression, character consistency, causal dependence, commonsense plausibility, and narrative interpretation.

## 1. Taxonomy of story reasoning resources

A useful distinction in this area is between **dataset papers** and **modeling papers that repurpose existing corpora**. Several influential papers are directly relevant to story reasoning while explicitly not introducing a new dataset: commonsense-grounded WritingPrompts generation [1908.09451], CAST-style adjacent-sentence commonsense consistency over ROCStories, Writing Prompts, and fairy tales [2105.01311], and narratology-grounded MovieSum/BookSum summarization [2605.03244]. This suggests that “story reasoning dataset” often names a benchmark role rather than a single release artifact.

The main benchmark families can be summarized as follows.

| Family | Representative resources | Core supervision |
|---|---|---|
| Synthetic interactive reasoning | e-QRAQ, Possible Stories, TimeTravel | clarification turns, multiple plausible endings, counterfactual rewrites |
| Long-form text and state tracking | WritingPrompts, STORIUM, NCP book corpus | prompt-story pairs, scene metadata, chapter-level continuation |
| Visual and audio storytelling | Extended MUGEN, FlintstonesSV, VIST-E, LSMDC-E, SoS | frame/sentence sequences, ending-image grounding, audio-story alignment |
| Movie and TV story understanding | TVQA, SyMoN, M-SYMON, SFD, FriendsQA | QA, clip-sentence alignment, long-video narrative understanding |
| Evaluation and reception | StoryER, SocialStoryFrames, EvolvR corpus | preference judgments, aspect ratings, comments, reader-response inferences |

These families differ in what they treat as the latent reasoning object. In some datasets the object is a **possible-answer set** or a **relevant variable**; in others it is a **counterfactual branch**, a **scene-level narrative state**, a **clip-sentence alignment**, or a **community-conditioned interpretation** [1708.01776][1909.04076][2010.01717][2406.13092][2512.15925].

## 2. Synthetic, interactive, and counterfactual benchmarks

The clearest text-only story reasoning benchmarks are the synthetic and semi-synthetic resources that make uncertainty or intervention explicit. **e-QRAQ** is a synthetic multi-turn dataset and simulator in which an agent reads an ambiguous short story, asks for hidden variable values, answers a challenge question, and receives explanations about why its query or answer was useful or not [1708.01776]. Its defining internal reasoning objects are the **set of possible answers** and the **set of relevant variables**, and the simulator exposes both through explanatory feedback. This makes e-QRAQ unusually suitable for research on sequential information acquisition, epistemic state tracking, and explanation generation under ambiguity.

**Possible Stories** moves from hidden-variable disambiguation to scenario-sensitive commonsense selection [2209.07760]. It contains **1,313 passages** and **4,533 multiple-choice questions**, built so that the same short story context is paired with four plausible endings and multiple questions, each of which makes a different ending the uniquely best answer. Its **consistency** metric measures whether a model answers all questions tied to the same context correctly. The dataset is notable for emphasizing conditional and counterfactual reasoning: the paper reports that **68.6%** of questions require counterfactual reasoning, far above comparison datasets. This design turns story completion into reasoning over a space of plausible futures rather than a single coherent ending.

**TimeTravel** makes counterfactual intervention the primary task [1909.04076]. Built on five-sentence ROCStories, it contains **29,849 counterfactual rewritings** and **80,115 counterfactual branches without a rewritten storyline**. The formal task is **Counterfactual Story Rewriting**: replace the second sentence with a counterfactual event and minimally rewrite the remaining ending so that it becomes coherent under the intervention. This setup directly targets causal dependence, temporal consistency, and counterfactual invariance, because the model must determine which downstream details change and which should remain unchanged.

Taken together, these benchmarks define a strong core of story reasoning datasets in the narrow sense: they expose uncertainty, alternative futures, or interventions as first-class supervision targets.

## 3. Long-form text corpora and latent narrative-state benchmarks

A second major strand consists of long-form corpora in which reasoning is not fully labeled but is structurally required. **WritingPrompts**, in the commonsense-grounding work of Mao et al., is used as a large story-generation corpus with **272K stories**, but the paper explicitly emphasizes that it provides only prompt-story supervision and no explicit labels for causal reasoning, physical plausibility, entity state tracking, or commonsense violations [1908.09451]. That absence is itself informative: it motivates augmenting a story corpus with external reasoning supervision such as SWAG, Story Cloze, and synthetic human-vs-machine ranking. A plausible implication is that large story corpora alone do not supply the supervision needed for commonsense-grounded narrative continuation.

**STORIUM** occupies a different position because it embeds naturally occurring narrative scaffolding into the corpus itself [2010.01717]. The released dataset contains **5,743 stories**, **25,092 scenes**, **448,264 scene entries**, and **126,041,738 total tokens**. Stories are written under ongoing narrative constraints using character descriptions, goals, strengths, weaknesses, items, scene introductions, locations, and challenges. The paper reports that in a small annotation study of **235 scene entries**, **77%** reference played cards and **80%** address the current challenge. This matters because the metadata is not decorative; it participates in continuation generation and thus supports character-grounded continuation, challenge-aware local planning, and long-context state tracking.

The long-form book setting in **Next-Chapter Prediction** pushes this further into chapter-scale reasoning [2503.22828]. The corpus is private because of copyright restrictions, but the benchmark design is explicit: **30 fiction books** are split into **22 train, 4 validation, and 4 test books**, yielding **1,004 train**, **162 validation**, and **181 test** chapter-level examples. Each input includes a **Global Story Sketch**, **Previous Story Summary**, **Character Sheets**, **Previous Chapter**, and **Next Chapter Synopsis**. The model is then asked to generate a reasoning trace ending in a detailed plan for the next chapter. This suggests a benchmark formulation in which story reasoning is measured by whether a generated plan improves the likelihood of the gold next chapter.

Some corpora in this family are only indirectly relevant. **Mythos**, for example, is a dataset of **590 stories from 64 authors across 5 distinct sources**, designed for personalized story generation through author-style inference rather than explicit narrative reasoning [2502.13028]. Its Author Writing Sheet organizes author tendencies into **Plot**, **Creativity**, **Development**, and **Language Use**, which can support weakly structured narrative analysis, but it is not a canonical reasoning benchmark.

## 4. Visual and audio story reasoning

Visual storytelling benchmarks extend story reasoning into reference resolution, multimodal grounding, and sequence-level consistency. **Make-A-Story** explicitly reframes story visualization as reasoning over references and co-references rather than framewise captioning [2211.13319]. The paper extends **MUGEN** by adding two new characters and expanding the background set from **2** to **6**, producing “Extended MUGEN” with **3** characters, **6** backgrounds, and an average of **3** references per story. It also modifies **FlintstonesSV** and **PororoSV** by replacing named entities with pronouns where possible, increasing referential difficulty. This design makes character continuity, background persistence, and pronoun resolution central to the benchmark.

A complementary multimodal task appears in **image-guided story ending generation** [2301.11357]. Here the relevant datasets are **VIST-E**, with **39,920 training**, **4,963 validation**, and **5,030 test** samples, and **LSMDC-E**, with **20,151 training**, **1,477 validation**, and **2,005 test** samples. Each instance supplies a four-sentence plot and an ending image, and the task is to generate the ending sentence. The benchmark is reasoning-sensitive because the ending image may reveal situation details not stated in the text, while the plot provides causal and temporal constraints not directly visible in the image.

**Sound of Story (SoS)** broadens the modality set again by adding non-speech audio to storytelling [2310.19264]. SoS contains **27,354 stories**, **535,707 total images**, and **984 hours of audio**, with speech-decoupled background sound aligned to image and text sequences. It is not an explicit reasoning dataset in the narrow sense, but it is relevant because it treats ambience, environmental sound, and background music as contextual evidence for story understanding. This suggests a multimodal notion of story reasoning in which narrative interpretation is supported not only by text and vision but also by non-linguistic acoustic context.

## 5. Movie and TV story understanding datasets

Story reasoning in video shifts the emphasis from local event recognition to long-range storyline understanding. **TVQA** was framed as the largest and only publicly available dataset for video story question answering at the time of the character-aware relations paper [2005.08646]. It contains **21.8K video clips**, **152.5K QA pairs**, and over **460 hours of video** from six TV shows, with subtitles that include speaker names. Its question distribution includes **84.8K what**, **17.7K who**, **17.8K where**, **15.8K why**, and **13.6K how** questions. The benchmark is story-reasoning-relevant because answering often requires grounding named recurring characters in faces, dialogue, and relations rather than relying on generic scene features.

**SyMoN** and **M-SYMON** push this into synopsis-style movie narrative grounding. **SyMoN** contains **5,193 video summaries** totaling **869 hours**, with **683,611 sentences**, and uses recap videos whose narration contains high coverage of multimodal story events and abundant mental-state descriptions [2203.05711]. **M-SYMON** expands the same general idea into a multilingual benchmark with **13,166 movie summary videos**, **2,136 total video hours**, and **7 languages**, plus **480 videos** and **101.5 hours** of manual fine-grained clip-sentence alignment [2406.13092]. These datasets are important because they treat synopsis alignment as a foundational subproblem of story understanding rather than a simple retrieval task.

A more direct QA formulation appears in the **Short Film Dataset (SFD)**, introduced as a story-level video understanding benchmark over **1,078 unique short films**, **243 hours** of video, and **4,885 MCQs and corresponding OEQs** [2406.10221]. Questions are organized into **setting**, **character**, **story**, and **theme** categories, and the benchmark is explicitly **test-only**. The paper’s temporal-window study shows consistent gains when models move from shot-level to full-movie context, supporting the claim that the dataset requires integration across multiple scenes and events.

**FriendsQA** specializes deep video understanding to one long-running story world [2412.17022]. It contains **35,222 single-episode questions** and **9,470 cross-episode questions**, for a total of over **44.6K questions**, across **234 episodes** of *Friends* with average episode length **1,358 seconds**. Its **14 fine-grained topics** combine **perception** and **inference** with **character**, **action**, and **location** categories and their combinations. This is one of the clearest benchmarks for storylines as long-range evolvement of characters, actions, and locations.

## 6. Evaluation datasets, reader-response inference, and reasoning about stories

A separate but increasingly important meaning of “story reasoning dataset” concerns evaluation and interpretation rather than continuation alone. **StoryER** is a story evaluation benchmark with explicit ranking, rating, and reasoning tasks [2210.08459]. Its ranking component is repeatedly described as “100k story ranking data,” while the split table sums to **116,971 pairs**; its aspect-comment resource contains **45,948 comments** over **12,669 unique stories**. The three tasks are: predicting a preference score, predicting aspect-specific ratings and confidences, and generating comments that explain those ratings. This is reasoning in an evaluative sense: the benchmark asks models not only to score stories but also to justify judgments about opening, ending, character shaping, scene description, and genre-specific effects.

**SocialStoryFrames** moves story reasoning toward narrative interpretation and reception [2512.15925]. The framework defines a **10-dimensional taxonomy** of reader response and applies it to **SSF-Corpus**, a curated collection of **6,140 social media stories** embedded in conversational and community context. The dimensions include **Overall Goal**, **Narrative Intent**, **Author Emotional Response**, **Character Appraisal**, **Causal Explanation**, **Prediction**, **Stance**, **Moral**, **Narrative Feeling**, and **Aesthetic Feeling**. This is not classical plot QA; it is contextualized inference about how many readers from a particular community would interpret, evaluate, and feel about a story.

**EvolvR** adds a synthetic reasoning-corpus perspective to story evaluation [2508.06046]. Using **StoryER**, **HANNA**, and **OpenMEVA**, it transforms pointwise human ratings into pairwise story-comparison supervision and synthesizes score-aligned CoTs. The pipeline starts from **80,000 stratified story pairs**, expands them into **800,000 candidate CoTs**, and filters them down to **536,177 high-quality rationales**. This is not a new human-annotated benchmark, but it is a substantial derived story-reasoning corpus for evaluator training. A plausible implication is that pairwise comparative reasoning may be a more scalable supervision format for open-ended story evaluation than direct pointwise scoring.

## 7. Recurring limitations and open directions

Several limitations recur across the benchmark landscape. First, local commonsense and global narrative coherence are not identical. The commonsense-grounded WritingPrompts work explicitly notes that **“story-level coherence is unlikely to be achieved using this method alone”**, even when SWAG and Story Cloze performance improves substantially [1908.09451]. This cautions against equating better short-horizon plausibility with full story reasoning.

Second, many of the richest corpora provide only indirect supervision. STORIUM’s metadata is natural language rather than canonical symbolic state, and the paper notes that community format and genre “worlds” may introduce domain biases [2010.01717]. SyMoN and SFD both show that subtitles or recap narration can be stronger cues than raw video, which suggests that some multimodal benchmarks are partly story-text understanding tasks rather than purely visual reasoning tasks [2203.05711][2406.10221].

Third, accessibility and release constraints remain significant. NCP’s 30-book corpus is private because of copyright restrictions [2503.22828]. Mythos releases are constrained by copyright and crawling policy, with links and identifiers rather than a fully redistributable corpus [2502.13028]. SFD is explicitly test-only [2406.10221]. These constraints complicate direct comparison and large-scale reproducibility.

Fourth, some influential papers are best understood as **benchmark reinterpretations** rather than dataset releases. CAST explicitly does not introduce a new story reasoning dataset or benchmark, but repurposes ROCStories, Writing Prompts, and fairy tales for adjacent-sentence commonsense consistency [2105.01311]. Likewise, the narratology-grounded screenplay summarization work does not create a new story reasoning dataset, but treats MovieSum and BookSum as reasoning-sensitive testbeds for plot nuclei, satellites, and character trajectories [2605.03244]. This suggests that future progress may depend as much on better task formulation and evaluation criteria as on new raw corpora.

Overall, the field is moving from short, one-shot plausibility tasks toward richer formulations of story reasoning: interactive ambiguity resolution, counterfactual branching, long-context narrative state tracking, multilingual synopsis grounding, deep video understanding over storylines, and community-conditioned reader-response inference. The common trend is away from isolated next-step prediction and toward representations that make character, context, causality, and interpretation explicit.

Source: https://www.emergentmind.com/topics/story-reasoning-dataset