StoryVideoQA: Narrative Video QA Benchmark

• StoryVideoQA is a large-scale benchmark for story-centric video question answering that models narrative evolution over long videos.
• It integrates characters, actions, dialogue, and spatial-temporal dynamics to capture long-range dependencies in storytelling.
• Recent research highlights challenges such as character grounding, temporal localization, and causal reasoning, guiding future improvements.

StoryVideoQA denotes a line of video question answering research oriented toward video story understanding rather than local fact extraction from short clips. In this setting, the target is not merely to recognize objects or actions in isolated moments, but to model storylines built from named characters, actions, locations, dialogue, and their long-range evolution across shots, scenes, episodes, or movies. The term is used both as a general designation for story-centric VideoQA and, more recently, as the title of a large-scale benchmark for deep video understanding (Choi et al., 2020, Wu et al., 4 Jun 2026).

1. From factoid VideoQA to story-centric reasoning

Early and contemporary papers draw a clear distinction between conventional VideoQA and story-centric QA. Factoid VideoQA typically concerns short clips and local questions such as object properties, counts, or single actions. StoryVideoQA instead targets temporal coherence, character relations, causality, emotions and behaviors, narrative flow, and long-range dependencies over coherent video narratives (Choi et al., 2020). Movie-oriented work framed this explicitly as understanding stories in movies through question answering over video and text, with questions ranging from simpler “Who” did “What” to “Whom”, to “Why” and “How” certain events occurred (Tapaswi et al., 2015).

This shift is especially visible in deep video understanding formulations for story videos. Recent benchmark work characterizes the core of DVU as understanding storylines with long-range evolvement, composed of characters, actions, locations, and their combinations, while also distinguishing perception questions from inference questions (Wu et al., 2024, Wu et al., 4 Jun 2026). A common simplification is to treat StoryVideoQA as merely longer short-clip QA; the literature instead indicates that named entities, dialogue grounding, scene continuity, and plot-level reasoning are central problem variables, not optional extensions (Geng et al., 2020, Wu et al., 4 Jun 2026).

DramaQA adds a further conceptual refinement by treating QA not only as supervision but as a graded test of understanding, organized into four difficulty levels aligned with Piagetian cognitive stages, from simple recall on a shot to causal reasoning over scenes (Choi et al., 2020). This developmental view has become one of the more distinctive ideas in StoryVideoQA evaluation.

2. Benchmark evolution and representative resources

The benchmark trajectory runs from movie-scale story comprehension, to cartoon and TV-story QA, to character-centered and topic-balanced long-video benchmarks. MovieQA established a multiple-choice story question answering setting over 408 movies and 14,944 questions, with multiple sources of information including video clips, plots, subtitles, scripts, and DVS; for the aligned video subset it reports 140 movies, 6,462 QAs, and an average clip duration of 202.7 seconds (Tapaswi et al., 2015). DeepStory introduced PororoQA, built from 20.5-hour cartoon videos with 16,066 scene-dialogue pairs, 27,328 scene descriptions, and 8,913 story-related QA pairs, and paired it with Deep Embedded Memory Networks (Kim et al., 2017).

DramaQA moved the field toward story-centric, character-centered, cognitively graded evaluation by using the TV drama “Another Miss Oh”, with 23,928 clips, 17,983 multiple-choice QA pairs, and 217,308 annotated images, plus difficulty labels and rich character annotations (Choi et al., 2020). TV-centric work then emphasized long clips with recurring characters and dialogue, especially through TVQA and TVQA+, which became the dominant VSQA setting for character-aware and relation-aware models (Geng et al., 2020, Li et al., 2021).

More recent datasets make the story structure itself the evaluation object. FriendsQA uses Friends episodes of average length 1,358 seconds and contains 44.6K questions evenly distributed across 14 fine-grained topics (Wu et al., 2024). StoryVideoQA scales this to over 363K QAs on 393.2 hours of diverse story videos, including TV series with average length 1,635 seconds and movies with average length 7,878 seconds, and describes itself as the largest DVU dataset to date (Wu et al., 4 Jun 2026).

Resource	Scale	Distinctive design
MovieQA (Tapaswi et al., 2015)	14,944 QAs, 408 movies	Multi-source story QA over video, plots, subtitles, scripts, DVS
PororoQA / DeepStory (Kim et al., 2017)	20.5 h, 8,913 QAs	Scene-dialogue story units and long-term memory
DramaQA (Choi et al., 2020)	17,983 QAs, 23,928 clips	Hierarchical difficulty and character-centered annotations
FriendsQA (Wu et al., 2024)	44.6K QAs	14 fine-grained story topics on long sitcom episodes
StoryVideoQA (Wu et al., 4 Jun 2026)	363K QAs, 393.2 h	Multi-genre DVU benchmark with StoryMindv2 and difficulty scores

3. Core representational primitives

A defining property of StoryVideoQA is the treatment of characters as first-class entities. DramaQA aligns character names across QA text, scripts, and visual metadata, and provides per-frame face and full-body boxes, behavior labels, emotion labels, speaker tags, and coreference-resolved scripts, thereby supporting cross-modal character tracking and local discourse coherence (Choi et al., 2020). “Character Matters” makes the same point more explicitly: without identifying the connection between appearing people and character names, a model is not able to obtain a genuine understanding of plots, so it introduces weakly supervised face naming and character-aware relation representations such as $$\langle \text{Lily}, \text{holding}, \text{flower} \rangle$$ (Geng et al., 2020).

A second recurrent primitive is the move from holistic frame embeddings to objects, relations, and events. Early contextual QA work represented the whole video as a heterogeneous, time-evolving graph $G(E,V,T,time)$ built from scene graphs, with timestamps attached to nodes and edges to support spatial, attribute, and temporal queries (Ganesan et al., 2018). Object-centric representation learning later turned a video into an evolving relational graph of tracked objects, then summarized each object’s “life” into a “resume” for deliberative relational reasoning (Dang et al., 2021). Relation-aware hierarchical attention similarly used detected objects, spatial graphs, semantic graphs, and subtitles to model static and dynamic object relations over time (Li et al., 2021).

A third primitive is the explicit modeling of story elements and their combinations. FriendsQA and StoryVideoQA formalize seven topic types—C, A, L, CA, CL, AL, and CAL—crossed with perception and inference, yielding 14 fine-grained categories that operationalize what it means to understand a storyline (Wu et al., 2024, Wu et al., 4 Jun 2026). This suggests a broad convergence in the field: story understanding is increasingly decomposed into entity identity, event structure, place, and cross-element interactions rather than treated as unstructured multimodal matching.

4. Modeling paradigms

The field has developed through several distinct paradigms. One line is symbolic and graph-based. “Video based Contextual Question Answering” parses dense captions into scene graphs, aggregates them over frames into $G(E,V,T,time)$, and answers contextual and temporal questions by subgraph matching and timestamp comparison rather than by end-to-end neural scoring (Ganesan et al., 2018). This approach is highly interpretable but limited in scalability and narrative semantics.

A second line is memory-based story reconstruction. DeepStory’s Deep Embedded Memory Networks reconstruct each video story unit by combining scene embeddings, dialogue, and retrieved descriptions, store these story sentences in long-term memory, and perform story selection and answer selection with attention-based BiLSTM matching (Kim et al., 2017). This formulation is one of the earliest direct templates for StoryVideoQA: define a story as a sequence of multimodal units, store it in memory, and answer by retrieving the relevant unit.

A third line is hierarchical, character-aware, and object-aware neural reasoning. DramaQA’s Multi-level Context Matching model combines script and visual streams at low and high levels, guided by character queries extracted from the QA text (Choi et al., 2020). Character Matters adds weakly supervised face naming, character-aware object augmentation, and character-aware visual relations into a Transformer-based reasoning network (Geng et al., 2020). Object-Centric Representation Learning represents tracked objects as temporal parts and object resumes (Dang et al., 2021), while Relation-aware Hierarchical Attention uses graph attention over object relations and hierarchical multimodal attention on TVQA+ (Li et al., 2021). Event-Correlated GNNs add a dense-caption modality to represent temporally localized events and perform cross-modal reasoning over caption, video, and question graphs (Yin et al., 2023).

Recent work on long-video settings extends these ideas into selection and agentic reasoning. Query-based frame selection replaces uniform sampling with submodular mutual information criteria and reports an accuracy improvement of up to 4% on MVBench over uniform sampling (Patil et al., 12 Jan 2026). SVAgent introduces a storyline-guided cross-modal multi-agent framework in which a storyline agent progressively constructs a narrative representation, textual and visual decision agents answer under storyline guidance, and a meta-agent aligns their outputs (Yang et al., 6 Apr 2026). PlotTree reorganizes long-range video content into a hierarchical plot structure and then performs retrieval-augmented QA over plot nodes (Wu et al., 4 Jun 2026). Taken together, these works suggest a shift from flat frame sequences toward narrative abstractions, structured memories, and explicit storyline control.

5. Evaluation protocols and empirical behavior

Multiple-choice accuracy remains the dominant metric, but StoryVideoQA work increasingly adds structure to evaluation. DramaQA computes per-level accuracy for Diff. 1–4, overall accuracy, and Diff. Avg., with the full MCM model reaching 75.96, 74.65, 57.36, and 56.63 on Diff. 1–4 respectively, 71.14 overall, and 66.15 Diff. Avg. on the test set (Choi et al., 2020). The performance drop from Diff. 1–2 to Diff. 3–4 is central: shot-based recall is substantially easier than scene-based temporal or causal reasoning.

FriendsQA and StoryVideoQA introduce explicit difficulty formulas. FriendsQA defines $\sigma_l^i = |V_w^i|/|V_r^i|$, $\sigma_c^i = |C_w^i|/|C_r^i|$, and $\sigma^i = \sigma_l^i/\mu_l + \sigma_c^i/\mu_c$, then bins questions into easy, medium, and hard (Wu et al., 2024). StoryVideoQA expands this into a three-part measure with question difficulty $D_q$, answer difficulty $D_a$, and question-answer difficulty $D_{qa}$, combined as

$$D(\{q,\{a_i\}_{i=1}^5\}) = \frac{D_q + D_a + D_{qa}}{3}.$$

Across models, accuracy decreases as these difficulty components increase (Wu et al., 4 Jun 2026).

Empirically, several patterns recur. First, long-range story understanding remains substantially harder than factoid QA: FriendsQA reports that VideoChat2 reaches 61.70% on NExT-QA but only 44.05% on FriendsQA (Wu et al., 2024). Second, character-centric reasoning is a consistent bottleneck: both FriendsQA and StoryVideoQA report that current models cannot fully maintain long-range character associations (Wu et al., 2024, Wu et al., 4 Jun 2026). Third, structure helps: on StoryVideoQA-G, PlotTree reaches 86.50% with Gemini-2.0-Flash and 88.80% with Gemini-3-Flash, exceeding Video2RAG and VideoTree under the same answering backbones (Wu et al., 4 Jun 2026). StoryVideoQA-GA, the anonymized variant in which characters and locations are replaced by generic placeholders, further shows that prior knowledge contributes nontrivially to performance, because many models drop when names and locations are masked (Wu et al., 4 Jun 2026).

6. Limitations, controversies, and future directions

The literature is consistent about present limitations. A major issue is character grounding: StoryVideoQA systems frequently confuse named characters, especially in large casts or across long videos, even when local visual evidence is adequate (Geng et al., 2020, Wu et al., 4 Jun 2026). Another is temporal localization: Story videos are long, relevant segments may be short, and perception questions can become harder than inference questions simply because the evidence is temporally sparse (Wu et al., 2024). A third is commonsense and causal reasoning: DramaQA notes failures when “why” questions require social norms or background objects beyond visible and scripted cues, and relation-aware TVQA work similarly observes difficulty on causal and counting questions (Choi et al., 2020, Li et al., 2021).

There are also resource and domain limitations. Rich character-centered annotations are labor-intensive, and single-domain datasets risk overfitting to recurring motifs, genres, or cultural norms (Choi et al., 2020). FriendsQA is restricted to Friends episodes, while StoryVideoQA, although multi-genre, still draws heavily from English-language and highly popular TV and film corpora (Wu et al., 2024, Wu et al., 4 Jun 2026). Another persistent concern is that models may answer from world knowledge rather than genuine video understanding, a problem made explicit by anonymized evaluation in StoryVideoQA-GA (Wu et al., 4 Jun 2026).

The main future directions are correspondingly clear. Benchmark construction is moving toward larger, multi-genre, automatically generated DVU corpora via multi-agent pipelines such as StoryMind and StoryMindv2 (Wu et al., 2024, Wu et al., 4 Jun 2026). Modeling is moving toward hierarchical plot structures, storyline-guided multi-agent reasoning, character libraries, and more efficient long-video retrieval (Yang et al., 6 Apr 2026, Wu et al., 4 Jun 2026). Several works also point to the need for audio cues, richer annotations of objects and places, longer-horizon character development, and multi-scale temporal structure from frame to scene to episode (Choi et al., 2020, Li et al., 2021). A plausible implication is that future StoryVideoQA systems will be judged less by flat accuracy alone and more by whether they can sustain coherent, grounded, and interpretable reasoning over evolving plots.