Story Reasoning Dataset
- Story Reasoning Dataset is a narrative benchmark encompassing task designs for plot progression, character consistency, causal dependence, and narrative interpretation.
- It integrates synthetic, interactive, and counterfactual benchmarks that require sequential information acquisition and state tracking in ambiguous narratives.
- The benchmarks also span long-form texts, visual storytelling, and evaluative frameworks to support multimodal grounding and community-conditioned interpretations.
Searching arXiv for recent and foundational papers on story reasoning datasets and related benchmarks. I’m checking arXiv entries relevant to story reasoning, narrative QA, and story generation benchmarks to ground the article in current literature. A story reasoning dataset is a narrative benchmark in which success depends on more than surface fluency or local lexical matching. Across the literature, such datasets operationalize reasoning through tasks such as interactive disambiguation, counterfactual rewriting, long-context continuation under evolving narrative state, multimodal alignment between stories and images or video, deep video understanding over storylines, and reader-response inference conditioned on social context (Rosenbaum et al., 2017, Akoury et al., 2020, Ghermi et al., 2024, Mire et al., 17 Dec 2025). Taken together, these resources suggest that story reasoning is not a single benchmark format but a family of task designs centered on plot progression, character consistency, causal dependence, commonsense plausibility, and narrative interpretation.
1. Taxonomy of story reasoning resources
A useful distinction in this area is between dataset papers and modeling papers that repurpose existing corpora. Several influential papers are directly relevant to story reasoning while explicitly not introducing a new dataset: commonsense-grounded WritingPrompts generation (Mao et al., 2019), CAST-style adjacent-sentence commonsense consistency over ROCStories, Writing Prompts, and fairy tales (Peng et al., 2021), and narratology-grounded MovieSum/BookSum summarization (Lu et al., 5 May 2026). This suggests that “story reasoning dataset” often names a benchmark role rather than a single release artifact.
The main benchmark families can be summarized as follows.
| Family | Representative resources | Core supervision |
|---|---|---|
| Synthetic interactive reasoning | e-QRAQ, Possible Stories, TimeTravel | clarification turns, multiple plausible endings, counterfactual rewrites |
| Long-form text and state tracking | WritingPrompts, STORIUM, NCP book corpus | prompt-story pairs, scene metadata, chapter-level continuation |
| Visual and audio storytelling | Extended MUGEN, FlintstonesSV, VIST-E, LSMDC-E, SoS | frame/sentence sequences, ending-image grounding, audio-story alignment |
| Movie and TV story understanding | TVQA, SyMoN, M-SYMON, SFD, FriendsQA | QA, clip-sentence alignment, long-video narrative understanding |
| Evaluation and reception | StoryER, SocialStoryFrames, EvolvR corpus | preference judgments, aspect ratings, comments, reader-response inferences |
These families differ in what they treat as the latent reasoning object. In some datasets the object is a possible-answer set or a relevant variable; in others it is a counterfactual branch, a scene-level narrative state, a clip-sentence alignment, or a community-conditioned interpretation (Rosenbaum et al., 2017, Qin et al., 2019, Akoury et al., 2020, Sun et al., 2024, Mire et al., 17 Dec 2025).
2. Synthetic, interactive, and counterfactual benchmarks
The clearest text-only story reasoning benchmarks are the synthetic and semi-synthetic resources that make uncertainty or intervention explicit. e-QRAQ is a synthetic multi-turn dataset and simulator in which an agent reads an ambiguous short story, asks for hidden variable values, answers a challenge question, and receives explanations about why its query or answer was useful or not (Rosenbaum et al., 2017). Its defining internal reasoning objects are the set of possible answers and the set of relevant variables, and the simulator exposes both through explanatory feedback. This makes e-QRAQ unusually suitable for research on sequential information acquisition, epistemic state tracking, and explanation generation under ambiguity.
Possible Stories moves from hidden-variable disambiguation to scenario-sensitive commonsense selection (Ashida et al., 2022). It contains 1,313 passages and 4,533 multiple-choice questions, built so that the same short story context is paired with four plausible endings and multiple questions, each of which makes a different ending the uniquely best answer. Its consistency metric measures whether a model answers all questions tied to the same context correctly. The dataset is notable for emphasizing conditional and counterfactual reasoning: the paper reports that 68.6% of questions require counterfactual reasoning, far above comparison datasets. This design turns story completion into reasoning over a space of plausible futures rather than a single coherent ending.
TimeTravel makes counterfactual intervention the primary task (Qin et al., 2019). Built on five-sentence ROCStories, it contains 29,849 counterfactual rewritings and 80,115 counterfactual branches without a rewritten storyline. The formal task is Counterfactual Story Rewriting: replace the second sentence with a counterfactual event and minimally rewrite the remaining ending so that it becomes coherent under the intervention. This setup directly targets causal dependence, temporal consistency, and counterfactual invariance, because the model must determine which downstream details change and which should remain unchanged.
Taken together, these benchmarks define a strong core of story reasoning datasets in the narrow sense: they expose uncertainty, alternative futures, or interventions as first-class supervision targets.
3. Long-form text corpora and latent narrative-state benchmarks
A second major strand consists of long-form corpora in which reasoning is not fully labeled but is structurally required. WritingPrompts, in the commonsense-grounding work of Mao et al., is used as a large story-generation corpus with 272K stories, but the paper explicitly emphasizes that it provides only prompt-story supervision and no explicit labels for causal reasoning, physical plausibility, entity state tracking, or commonsense violations (Mao et al., 2019). That absence is itself informative: it motivates augmenting a story corpus with external reasoning supervision such as SWAG, Story Cloze, and synthetic human-vs-machine ranking. A plausible implication is that large story corpora alone do not supply the supervision needed for commonsense-grounded narrative continuation.
STORIUM occupies a different position because it embeds naturally occurring narrative scaffolding into the corpus itself (Akoury et al., 2020). The released dataset contains 5,743 stories, 25,092 scenes, 448,264 scene entries, and 126,041,738 total tokens. Stories are written under ongoing narrative constraints using character descriptions, goals, strengths, weaknesses, items, scene introductions, locations, and challenges. The paper reports that in a small annotation study of 235 scene entries, 77% reference played cards and 80% address the current challenge. This matters because the metadata is not decorative; it participates in continuation generation and thus supports character-grounded continuation, challenge-aware local planning, and long-context state tracking.
The long-form book setting in Next-Chapter Prediction pushes this further into chapter-scale reasoning (Gurung et al., 28 Mar 2025). The corpus is private because of copyright restrictions, but the benchmark design is explicit: 30 fiction books are split into 22 train, 4 validation, and 4 test books, yielding 1,004 train, 162 validation, and 181 test chapter-level examples. Each input includes a Global Story Sketch, Previous Story Summary, Character Sheets, Previous Chapter, and Next Chapter Synopsis. The model is then asked to generate a reasoning trace ending in a detailed plan for the next chapter. This suggests a benchmark formulation in which story reasoning is measured by whether a generated plan improves the likelihood of the gold next chapter.
Some corpora in this family are only indirectly relevant. Mythos, for example, is a dataset of 590 stories from 64 authors across 5 distinct sources, designed for personalized story generation through author-style inference rather than explicit narrative reasoning (Kumar et al., 18 Feb 2025). Its Author Writing Sheet organizes author tendencies into Plot, Creativity, Development, and Language Use, which can support weakly structured narrative analysis, but it is not a canonical reasoning benchmark.
4. Visual and audio story reasoning
Visual storytelling benchmarks extend story reasoning into reference resolution, multimodal grounding, and sequence-level consistency. Make-A-Story explicitly reframes story visualization as reasoning over references and co-references rather than framewise captioning (Rahman et al., 2022). The paper extends MUGEN by adding two new characters and expanding the background set from 2 to 6, producing “Extended MUGEN” with 3 characters, 6 backgrounds, and an average of 3 references per story. It also modifies FlintstonesSV and PororoSV by replacing named entities with pronouns where possible, increasing referential difficulty. This design makes character continuity, background persistence, and pronoun resolution central to the benchmark.
A complementary multimodal task appears in image-guided story ending generation (Zhou et al., 2023). Here the relevant datasets are VIST-E, with 39,920 training, 4,963 validation, and 5,030 test samples, and LSMDC-E, with 20,151 training, 1,477 validation, and 2,005 test samples. Each instance supplies a four-sentence plot and an ending image, and the task is to generate the ending sentence. The benchmark is reasoning-sensitive because the ending image may reveal situation details not stated in the text, while the plot provides causal and temporal constraints not directly visible in the image.
Sound of Story (SoS) broadens the modality set again by adding non-speech audio to storytelling (Bae et al., 2023). SoS contains 27,354 stories, 535,707 total images, and 984 hours of audio, with speech-decoupled background sound aligned to image and text sequences. It is not an explicit reasoning dataset in the narrow sense, but it is relevant because it treats ambience, environmental sound, and background music as contextual evidence for story understanding. This suggests a multimodal notion of story reasoning in which narrative interpretation is supported not only by text and vision but also by non-linguistic acoustic context.
5. Movie and TV story understanding datasets
Story reasoning in video shifts the emphasis from local event recognition to long-range storyline understanding. TVQA was framed as the largest and only publicly available dataset for video story question answering at the time of the character-aware relations paper (Geng et al., 2020). It contains 21.8K video clips, 152.5K QA pairs, and over 460 hours of video from six TV shows, with subtitles that include speaker names. Its question distribution includes 84.8K what, 17.7K who, 17.8K where, 15.8K why, and 13.6K how questions. The benchmark is story-reasoning-relevant because answering often requires grounding named recurring characters in faces, dialogue, and relations rather than relying on generic scene features.
SyMoN and M-SYMON push this into synopsis-style movie narrative grounding. SyMoN contains 5,193 video summaries totaling 869 hours, with 683,611 sentences, and uses recap videos whose narration contains high coverage of multimodal story events and abundant mental-state descriptions (Sun et al., 2022). M-SYMON expands the same general idea into a multilingual benchmark with 13,166 movie summary videos, 2,136 total video hours, and 7 languages, plus 480 videos and 101.5 hours of manual fine-grained clip-sentence alignment (Sun et al., 2024). These datasets are important because they treat synopsis alignment as a foundational subproblem of story understanding rather than a simple retrieval task.
A more direct QA formulation appears in the Short Film Dataset (SFD), introduced as a story-level video understanding benchmark over 1,078 unique short films, 243 hours of video, and 4,885 MCQs and corresponding OEQs (Ghermi et al., 2024). Questions are organized into setting, character, story, and theme categories, and the benchmark is explicitly test-only. The paper’s temporal-window study shows consistent gains when models move from shot-level to full-movie context, supporting the claim that the dataset requires integration across multiple scenes and events.
FriendsQA specializes deep video understanding to one long-running story world (Wu et al., 2024). It contains 35,222 single-episode questions and 9,470 cross-episode questions, for a total of over 44.6K questions, across 234 episodes of Friends with average episode length 1,358 seconds. Its 14 fine-grained topics combine perception and inference with character, action, and location categories and their combinations. This is one of the clearest benchmarks for storylines as long-range evolvement of characters, actions, and locations.
6. Evaluation datasets, reader-response inference, and reasoning about stories
A separate but increasingly important meaning of “story reasoning dataset” concerns evaluation and interpretation rather than continuation alone. StoryER is a story evaluation benchmark with explicit ranking, rating, and reasoning tasks (Chen et al., 2022). Its ranking component is repeatedly described as “100k story ranking data,” while the split table sums to 116,971 pairs; its aspect-comment resource contains 45,948 comments over 12,669 unique stories. The three tasks are: predicting a preference score, predicting aspect-specific ratings and confidences, and generating comments that explain those ratings. This is reasoning in an evaluative sense: the benchmark asks models not only to score stories but also to justify judgments about opening, ending, character shaping, scene description, and genre-specific effects.
SocialStoryFrames moves story reasoning toward narrative interpretation and reception (Mire et al., 17 Dec 2025). The framework defines a 10-dimensional taxonomy of reader response and applies it to SSF-Corpus, a curated collection of 6,140 social media stories embedded in conversational and community context. The dimensions include Overall Goal, Narrative Intent, Author Emotional Response, Character Appraisal, Causal Explanation, Prediction, Stance, Moral, Narrative Feeling, and Aesthetic Feeling. This is not classical plot QA; it is contextualized inference about how many readers from a particular community would interpret, evaluate, and feel about a story.
EvolvR adds a synthetic reasoning-corpus perspective to story evaluation (Wang et al., 8 Aug 2025). Using StoryER, HANNA, and OpenMEVA, it transforms pointwise human ratings into pairwise story-comparison supervision and synthesizes score-aligned CoTs. The pipeline starts from 80,000 stratified story pairs, expands them into 800,000 candidate CoTs, and filters them down to 536,177 high-quality rationales. This is not a new human-annotated benchmark, but it is a substantial derived story-reasoning corpus for evaluator training. A plausible implication is that pairwise comparative reasoning may be a more scalable supervision format for open-ended story evaluation than direct pointwise scoring.
7. Recurring limitations and open directions
Several limitations recur across the benchmark landscape. First, local commonsense and global narrative coherence are not identical. The commonsense-grounded WritingPrompts work explicitly notes that “story-level coherence is unlikely to be achieved using this method alone”, even when SWAG and Story Cloze performance improves substantially (Mao et al., 2019). This cautions against equating better short-horizon plausibility with full story reasoning.
Second, many of the richest corpora provide only indirect supervision. STORIUM’s metadata is natural language rather than canonical symbolic state, and the paper notes that community format and genre “worlds” may introduce domain biases (Akoury et al., 2020). SyMoN and SFD both show that subtitles or recap narration can be stronger cues than raw video, which suggests that some multimodal benchmarks are partly story-text understanding tasks rather than purely visual reasoning tasks (Sun et al., 2022, Ghermi et al., 2024).
Third, accessibility and release constraints remain significant. NCP’s 30-book corpus is private because of copyright restrictions (Gurung et al., 28 Mar 2025). Mythos releases are constrained by copyright and crawling policy, with links and identifiers rather than a fully redistributable corpus (Kumar et al., 18 Feb 2025). SFD is explicitly test-only (Ghermi et al., 2024). These constraints complicate direct comparison and large-scale reproducibility.
Fourth, some influential papers are best understood as benchmark reinterpretations rather than dataset releases. CAST explicitly does not introduce a new story reasoning dataset or benchmark, but repurposes ROCStories, Writing Prompts, and fairy tales for adjacent-sentence commonsense consistency (Peng et al., 2021). Likewise, the narratology-grounded screenplay summarization work does not create a new story reasoning dataset, but treats MovieSum and BookSum as reasoning-sensitive testbeds for plot nuclei, satellites, and character trajectories (Lu et al., 5 May 2026). This suggests that future progress may depend as much on better task formulation and evaluation criteria as on new raw corpora.
Overall, the field is moving from short, one-shot plausibility tasks toward richer formulations of story reasoning: interactive ambiguity resolution, counterfactual branching, long-context narrative state tracking, multilingual synopsis grounding, deep video understanding over storylines, and community-conditioned reader-response inference. The common trend is away from isolated next-step prediction and toward representations that make character, context, causality, and interpretation explicit.