---
title: 'StoryVideoQA: Narrative Video QA Benchmark'
url: https://www.emergentmind.com/topics/storyvideoqa
type: topic
---

# StoryVideoQA: Narrative Video QA Benchmark

StoryVideoQA denotes a line of video question answering research oriented toward **video story understanding** rather than local fact extraction from short clips. In this setting, the target is not merely to recognize objects or actions in isolated moments, but to model **storylines** built from named characters, actions, locations, dialogue, and their long-range evolution across shots, scenes, episodes, or movies. The term is used both as a general designation for story-centric VideoQA and, more recently, as the title of a large-scale benchmark for deep video understanding [2005.03356][2606.06338].

## 1. From factoid VideoQA to story-centric reasoning

Early and contemporary papers draw a clear distinction between conventional VideoQA and story-centric QA. Factoid VideoQA typically concerns short clips and local questions such as object properties, counts, or single actions. StoryVideoQA instead targets **temporal coherence**, **character relations**, **causality**, **emotions and behaviors**, **narrative flow**, and long-range dependencies over coherent video narratives [2005.03356]. Movie-oriented work framed this explicitly as understanding stories in movies through question answering over video and text, with questions ranging from simpler “Who” did “What” to “Whom”, to “Why” and “How” certain events occurred [1512.02902].

This shift is especially visible in deep video understanding formulations for story videos. Recent benchmark work characterizes the core of DVU as understanding storylines with long-range evolvement, composed of **characters**, **actions**, **locations**, and their combinations, while also distinguishing **perception** questions from **inference** questions [2412.17022][2606.06338]. A common simplification is to treat StoryVideoQA as merely longer short-clip QA; the literature instead indicates that named entities, dialogue grounding, scene continuity, and plot-level reasoning are central problem variables, not optional extensions [2005.08646][2606.06338].

DramaQA adds a further conceptual refinement by treating QA not only as supervision but as a **graded test of understanding**, organized into four difficulty levels aligned with Piagetian cognitive stages, from simple recall on a shot to causal reasoning over scenes [2005.03356]. This developmental view has become one of the more distinctive ideas in StoryVideoQA evaluation.

## 2. Benchmark evolution and representative resources

The benchmark trajectory runs from movie-scale story comprehension, to cartoon and TV-story QA, to character-centered and topic-balanced long-video benchmarks. MovieQA established a multiple-choice story question answering setting over **408 movies** and **14,944 questions**, with multiple sources of information including video clips, plots, subtitles, scripts, and DVS; for the aligned video subset it reports **140 movies**, **6,462 QAs**, and an average clip duration of **202.7 seconds** [1512.02902]. DeepStory introduced PororoQA, built from **20.5-hour** cartoon videos with **16,066** scene-dialogue pairs, **27,328** scene descriptions, and **8,913** story-related QA pairs, and paired it with Deep Embedded Memory Networks [1707.00836].

DramaQA moved the field toward story-centric, character-centered, cognitively graded evaluation by using the TV drama **“Another Miss Oh”**, with **23,928 clips**, **17,983 multiple-choice QA pairs**, and **217,308 annotated images**, plus difficulty labels and rich character annotations [2005.03356]. TV-centric work then emphasized long clips with recurring characters and dialogue, especially through TVQA and TVQA+, which became the dominant VSQA setting for character-aware and relation-aware models [2005.08646][2105.06160].

More recent datasets make the story structure itself the evaluation object. FriendsQA uses *Friends* episodes of average length **1,358 seconds** and contains **44.6K questions** evenly distributed across **14 fine-grained topics** [2412.17022]. StoryVideoQA scales this to **over 363K QAs** on **393.2 hours** of diverse story videos, including TV series with average length **1,635 seconds** and movies with average length **7,878 seconds**, and describes itself as the largest DVU dataset to date [2606.06338].

| Resource | Scale | Distinctive design |
|---|---:|---|
| MovieQA [1512.02902] | 14,944 QAs, 408 movies | Multi-source story QA over video, plots, subtitles, scripts, DVS |
| PororoQA / DeepStory [1707.00836] | 20.5 h, 8,913 QAs | Scene-dialogue story units and long-term memory |
| DramaQA [2005.03356] | 17,983 QAs, 23,928 clips | Hierarchical difficulty and character-centered annotations |
| FriendsQA [2412.17022] | 44.6K QAs | 14 fine-grained story topics on long sitcom episodes |
| StoryVideoQA [2606.06338] | 363K QAs, 393.2 h | Multi-genre DVU benchmark with StoryMindv2 and difficulty scores |

## 3. Core representational primitives

A defining property of StoryVideoQA is the treatment of **characters as first-class entities**. DramaQA aligns character names across QA text, scripts, and visual metadata, and provides per-frame face and full-body boxes, behavior labels, emotion labels, speaker tags, and coreference-resolved scripts, thereby supporting cross-modal character tracking and local discourse coherence [2005.03356]. “Character Matters” makes the same point more explicitly: without identifying the connection between appearing people and character names, a model is not able to obtain a genuine understanding of plots, so it introduces weakly supervised face naming and character-aware relation representations such as \(\langle \text{Lily}, \text{holding}, \text{flower} \rangle\) [2005.08646].

A second recurrent primitive is the move from holistic frame embeddings to **objects, relations, and events**. Early contextual QA work represented the whole video as a heterogeneous, time-evolving graph \(G(E,V,T,time)\) built from scene graphs, with timestamps attached to nodes and edges to support spatial, attribute, and temporal queries [1804.07399]. Object-centric representation learning later turned a video into an evolving relational graph of tracked objects, then summarized each object’s “life” into a “resume” for deliberative relational reasoning [2104.05166]. Relation-aware hierarchical attention similarly used detected objects, spatial graphs, semantic graphs, and subtitles to model static and dynamic object relations over time [2105.06160].

A third primitive is the explicit modeling of **story elements and their combinations**. FriendsQA and StoryVideoQA formalize seven topic types—C, A, L, CA, CL, AL, and CAL—crossed with perception and inference, yielding 14 fine-grained categories that operationalize what it means to understand a storyline [2412.17022][2606.06338]. This suggests a broad convergence in the field: story understanding is increasingly decomposed into entity identity, event structure, place, and cross-element interactions rather than treated as unstructured multimodal matching.

## 4. Modeling paradigms

The field has developed through several distinct paradigms. One line is **symbolic and graph-based**. “Video based Contextual Question Answering” parses dense captions into scene graphs, aggregates them over frames into \(G(E,V,T,time)\), and answers contextual and temporal questions by subgraph matching and timestamp comparison rather than by end-to-end neural scoring [1804.07399]. This approach is highly interpretable but limited in scalability and narrative semantics.

A second line is **memory-based story reconstruction**. DeepStory’s Deep Embedded Memory Networks reconstruct each video story unit by combining scene embeddings, dialogue, and retrieved descriptions, store these story sentences in long-term memory, and perform story selection and answer selection with attention-based BiLSTM matching [1707.00836]. This formulation is one of the earliest direct templates for StoryVideoQA: define a story as a sequence of multimodal units, store it in memory, and answer by retrieving the relevant unit.

A third line is **hierarchical, character-aware, and object-aware neural reasoning**. DramaQA’s Multi-level Context Matching model combines script and visual streams at low and high levels, guided by character queries extracted from the QA text [2005.03356]. Character Matters adds weakly supervised face naming, character-aware object augmentation, and character-aware visual relations into a Transformer-based reasoning network [2005.08646]. Object-Centric Representation Learning represents tracked objects as temporal parts and object resumes [2104.05166], while Relation-aware Hierarchical Attention uses graph attention over object relations and hierarchical multimodal attention on TVQA+ [2105.06160]. Event-Correlated GNNs add a dense-caption modality to represent temporally localized events and perform cross-modal reasoning over caption, video, and question graphs [2312.12721].

Recent work on long-video settings extends these ideas into **selection and agentic reasoning**. Query-based frame selection replaces uniform sampling with submodular mutual information criteria and reports an accuracy improvement of up to **4%** on MVBench over uniform sampling [2601.07459]. SVAgent introduces a storyline-guided cross-modal multi-agent framework in which a storyline agent progressively constructs a narrative representation, textual and visual decision agents answer under storyline guidance, and a meta-agent aligns their outputs [2604.05079]. PlotTree reorganizes long-range video content into a hierarchical plot structure and then performs retrieval-augmented QA over plot nodes [2606.06338]. Taken together, these works suggest a shift from flat frame sequences toward **narrative abstractions, structured memories, and explicit storyline control**.

## 5. Evaluation protocols and empirical behavior

Multiple-choice accuracy remains the dominant metric, but StoryVideoQA work increasingly adds structure to evaluation. DramaQA computes per-level accuracy for **Diff. 1–4**, overall accuracy, and **Diff. Avg.**, with the full MCM model reaching **75.96**, **74.65**, **57.36**, and **56.63** on Diff. 1–4 respectively, **71.14** overall, and **66.15** Diff. Avg. on the test set [2005.03356]. The performance drop from Diff. 1–2 to Diff. 3–4 is central: shot-based recall is substantially easier than scene-based temporal or causal reasoning.

FriendsQA and StoryVideoQA introduce explicit difficulty formulas. FriendsQA defines \(\sigma_l^i = |V_w^i|/|V_r^i|\), \(\sigma_c^i = |C_w^i|/|C_r^i|\), and \(\sigma^i = \sigma_l^i/\mu_l + \sigma_c^i/\mu_c\), then bins questions into easy, medium, and hard [2412.17022]. StoryVideoQA expands this into a three-part measure with question difficulty \(D_q\), answer difficulty \(D_a\), and question-answer difficulty \(D_{qa}\), combined as
\[
D(\{q,\{a_i\}_{i=1}^5\}) = \frac{D_q + D_a + D_{qa}}{3}.
\]
Across models, accuracy decreases as these difficulty components increase [2606.06338].

Empirically, several patterns recur. First, **long-range story understanding remains substantially harder than factoid QA**: FriendsQA reports that VideoChat2 reaches **61.70%** on NExT-QA but only **44.05%** on FriendsQA [2412.17022]. Second, **character-centric reasoning is a consistent bottleneck**: both FriendsQA and StoryVideoQA report that current models cannot fully maintain long-range character associations [2412.17022][2606.06338]. Third, structure helps: on StoryVideoQA-G, PlotTree reaches **86.50%** with Gemini-2.0-Flash and **88.80%** with Gemini-3-Flash, exceeding Video2RAG and VideoTree under the same answering backbones [2606.06338]. StoryVideoQA-GA, the anonymized variant in which characters and locations are replaced by generic placeholders, further shows that prior knowledge contributes nontrivially to performance, because many models drop when names and locations are masked [2606.06338].

## 6. Limitations, controversies, and future directions

The literature is consistent about present limitations. A major issue is **character grounding**: StoryVideoQA systems frequently confuse named characters, especially in large casts or across long videos, even when local visual evidence is adequate [2005.08646][2606.06338]. Another is **temporal localization**: Story videos are long, relevant segments may be short, and perception questions can become harder than inference questions simply because the evidence is temporally sparse [2412.17022]. A third is **commonsense and causal reasoning**: DramaQA notes failures when “why” questions require social norms or background objects beyond visible and scripted cues, and relation-aware TVQA work similarly observes difficulty on causal and counting questions [2005.03356][2105.06160].

There are also resource and domain limitations. Rich character-centered annotations are labor-intensive, and single-domain datasets risk overfitting to recurring motifs, genres, or cultural norms [2005.03356]. FriendsQA is restricted to *Friends* episodes, while StoryVideoQA, although multi-genre, still draws heavily from English-language and highly popular TV and film corpora [2412.17022][2606.06338]. Another persistent concern is that models may answer from world knowledge rather than genuine video understanding, a problem made explicit by anonymized evaluation in StoryVideoQA-GA [2606.06338].

The main future directions are correspondingly clear. Benchmark construction is moving toward **larger, multi-genre, automatically generated DVU corpora** via multi-agent pipelines such as StoryMind and StoryMindv2 [2412.17022][2606.06338]. Modeling is moving toward **hierarchical plot structures**, **storyline-guided multi-agent reasoning**, **character libraries**, and more efficient long-video retrieval [2604.05079][2606.06338]. Several works also point to the need for **audio cues**, richer annotations of objects and places, longer-horizon character development, and multi-scale temporal structure from frame to scene to episode [2005.03356][2105.06160]. A plausible implication is that future StoryVideoQA systems will be judged less by flat accuracy alone and more by whether they can sustain coherent, grounded, and interpretable reasoning over evolving plots.

Source: https://www.emergentmind.com/topics/storyvideoqa