StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Dataset

Abstract: Video question answering (VideoQA) aims to answer questions about given videos. While existing approaches excel on factoid VideoQA, they struggle with deep video understanding (DVU), which requires the comprehension of complex storylines. This challenge arises from the inherent long-range video content, multi-faceted question types, and instance-level story elements, all of which constrain the scale and diversity of manually constructed DVU datasets.These difficulties constrain the scale and diversity of manually-constructed DVU dataset. To address these, we previously introduced StoryMind to automatically construct DVU datasets with balanced fine-grained topics. Though it can generate high-quality question-answer pairs (QAs) for TV series, it suffers significant performance degradation when handling longer and more complex movies. In this paper, we further design StoryMindv2, an enhanced multi-agent collaboration framework to generate high-quality DVU datasets for both TV series and movies. By integrating a novel supervisor-guided generation mechanism and a refined multi-reviewer voting strategy, the framework is utilized to construct StoryVideoQA, the largest DVU dataset to date, featuring over 363K QAs on 393.2 hours diverse story videos including TV series (avg. 1,635 seconds) and movies (avg. 7,878 seconds). Comprehensive evaluations of 20 state-of-the-art VideoQA methods on this large-scale benchmark reveal that they cannot fully maintain long-range character associations or construct a coherent understanding of complex storylines. To bridge this gap, we propose PlotTree, a novel video understanding agent, re-organizing long-range video content into a hierarchical plot structure, enabling efficient storyline reasoning on StoryVideoQA. Project page: https://github.com/nercms-mmap/StoryVideoQA/

• The paper introduces a scalable DVU benchmark that uses automated QA generation and achieves balanced, fine-grained topic coverage.
• It presents StoryMindv2 for multi-agent QA generation and PlotTree, a hierarchical agent for long-range, narrative reasoning in videos.
• Empirical evaluations reveal significant performance gaps in existing VideoQA models, emphasizing the need for advanced narrative understanding.

StoryVideoQA: A Scalable Benchmark and Agentic Paradigm for Deep Video Understanding

Introduction

"StoryVideoQA: Scaling Deep Video Understanding with a Large-Scale, Multi-Genre and Auto-Generated Dataset" (2606.06338) addresses the critical gap in video question answering (VideoQA) research: moving beyond short, factoid tasks toward comprehensive deep video understanding (DVU) of long-range, narrative-rich videos encompassing TV series and movies. The work systematically advances the state of DVU along three axes: (1) it diagnoses the empirical failures of current VideoQA systems on complex story-centered tasks, (2) it automates the large-scale and balanced construction of fine-grained, multi-genre DVU datasets, and (3) it introduces a hierarchical agent, PlotTree, for long-range story reasoning. The scale and thoroughness of this work provide a new empirical foundation and methodological direction for high-level narrative video comprehension.

Figure 1: Factoid VideoQA versus DVU—StoryVideoQA focuses on the long-range, multi-entity, high-inference tasks largely absent in prior benchmarks.

Dataset Construction: StoryMindv2 and StoryVideoQA

Limitations of Previous DVU Benchmarks

Factoid datasets (e.g., ActivityNet-QA, NExT-QA) focus on short clips and simple perceptual questions, while existing DVU benchmarks (e.g., MovieQA, DramaQA, HLVU) remain limited in scale, coverage, and topic distribution. Manual construction of comprehensive DVU datasets is stymied by video length, annotation labor (especially in movies), and inherent imbalances in topic coverage, particularly for long-range inference and composite story elements.

Figure 2: Distribution of five datasets across 14 fine-grained topics reveals major imbalance in prior work—StoryVideoQA achieves balanced covering.

Automated Multi-Agent Dataset Generation

To circumvent manual bottlenecks, the authors propose StoryMindv2, an enhanced multi-agent QA generation and filtering framework. Key innovations over the previous StoryMind system include:

• Supervisor-Guided Generation: Supervisor agents validate and correct generative failures, leveraging a fault archive for targeted generator feedback, elevating both factual accuracy and question diversity.
• Refined Multi-Reviewer Voting: Three independent reviewers filter QAs via majority voting, raising recall without sacrificing precision, supporting scale without accumulating annotation artifacts.
• Dynamic Topic Tooling: Fine-grained topic balancing is enforced by dynamically deactivating over-represented tools during generation, ensuring even coverage across the combinatorial topic space (perception/inference $\times$ 3W elements).

  Figure 3: Accuracy in automatic QA generation—StoryMindv2 notably outperforms the original StoryMind on long and complex movie sources.

  

  Figure 4: Workflow of the StoryMindv2 multi-agent QA dataset generation pipeline.

Scale and Composition of StoryVideoQA

Leveraging StoryMindv2, StoryVideoQA is constructed as the largest DVU dataset to date: 363K QA pairs over 393.2 hours of video, covering 3 major TV series and 78 top-rated movies. Its distinguishing features:

• Video Lengths: TV episodes average 1,635s; movies 7,878s—orders of magnitude longer than previous standards.
• Topic Distribution: Full coverage of 14 fine-grained topic types (perception/inference with all 3W element combinations), empirically balanced as shown by Gini index and entropy analyses.
• Difficulty Calibration: Each QA pair is labeled by composite difficulty measures capturing segment length, entity span, distractor entropicity, and question/answer semantic gap.

  Figure 5: StoryVideoQA realizes balanced distribution across all fine-grained story topics—essential for comprehensive model assessment.

  

Figure 6: The dataset’s difficulty distribution is well-calibrated for both question and answer, enabling insightful error analysis of evaluation results.

Figure 7: The included Character Library further supports explicit entity grounding, facilitating character-centric DVU tasks.

Benchmark Evaluation and Empirical Findings

A comprehensive evaluation suite includes 20 state-of-the-art VideoQA models: VLMs, MLLMs, and agentic workflows. Results on StoryVideoQA expose several critical shortcomings of existing models:

• VLMs (e.g., Vid-TLDR, VIOLETv2): Performance saturates at <25% accuracy—essentially near random-guess on 5-way multiple-choice for long-range DVU.
• MLLMs (e.g., VideoLLaMA2/3, ViLAMP): Despite pretraining, open models reach only 76% on the easiest subsets; closed models (Gemini-3-Flash) approach 88% but still fail on fine-grained, long-context inferences.
• Difficulty Sensitivity: Model accuracy drops sharply with increased question/answer complexity or cross-entity reasoning, validating the utility of difficulty annotations for root-cause analysis.

  Figure 8: Robust decline in model accuracy from conventional factoid VideoQA (e.g., NExT-QA) to StoryVideoQA’s complex DVU benchmarks.

  

  Figure 9: MLLMs outperform VLMs, but performance remains far from optimal, especially in fine-grained topics requiring cross-character or temporal inference.

  

  Figure 10: Average performance reveals movie-based long-range inference remains especially challenging for all model families.

PlotTree: Hierarchical Agentic Reasoning for Narrative Videos

The paper introduces PlotTree, a hierarchical agent for long video reasoning, addressing two structural bottlenecks:

• Flat event/caption retrieval (as in VideoTree, Video2RAG) fails to exploit narrative structure and long-range dependencies.
• Current MLLMs lack mechanisms for explicit, persistent entity linking and long-horizon context integration.

PlotTree Methodology:

• Hierarchical Plot Construction: Videos are segmented into annotated keyframes, named entities are explicitly grounded using a custom character library, and captions are clustered into multi-level semantic nodes via K-means with temporal decay.
• Semantic Abstraction: Parent nodes are generated via condensation of child node summaries, recursively forming a PlotTree from micro (frame/dialogue) to macro (scene/arc) structures.
• RAG Reasoning: Upon query, relevant nodes are retrieved across tree levels (using Qwen3 embeddings), forming a context for LLM-based answer synthesis.

  Figure 11: PlotTree architecture showing hierarchical node generation, node clustering, and recursive abstraction.

  

  Figure 12: Detailed process of leaf node creation with explicit character, dialogue, and summary binding.

  

  Figure 13: Plot condensation cascades, maintaining both temporal and semantic coherence as abstraction level increases.

Performance:

PlotTree, when equipped with explicit character identification and tested using Gemini-3-Flash, matches or surpasses closed MLLMs on all metrics: 88.8% overall accuracy on the hardest split, outperforming both flat RAG agents and direct frame-input MLLMs. The gains are most pronounced in topics demanding long-range, multi-entity inference.

Theoretical and Practical Implications

Theoretical:

• The results challenge the adequacy of flat RAG-based or single-window MLLM approaches for narrative reasoning, instead supporting explicit hierarchical modeling and persistent entity tracking.
• The dynamic agentic paradigm demonstrates that structural information loss in caption pre-processing can be offset by rigorous hierarchical abstraction and targeted node retrieval.

Practical:

• StoryVideoQA, with balanced, difficulty-annotated, and fully-automated construction, lowers the barrier for large-scale fine-grained DVU research.
• PlotTree's modular agentic approach can be adapted as a backbone for other long-range, multi-modal narrative understanding tasks, e.g., TV/movie retrieval, summarization, and interactive video assistants.

Future Directions:

• The scale of StoryVideoQA enables robust pretraining of tailored MLLMs for narrative comprehension, potentially closing the gap with closed-source systems.
• Hybrid models leveraging both agentic abstraction and native video encoding (e.g., plug-in recurrent or memory-based MLLMs) can further address current limitations in cross-temporal reasoning and entity tracking.
• Progressive anonymization experiments indicate that true video understanding must be disentangled from memorization or world knowledge, signaling demand for benchmarks robust against such confounds.

Conclusion

The work establishes StoryVideoQA as the first auto-generated, large-scale, perfectly topic-balanced, and difficulty-labeled DVU benchmark for story-centric videos, resolving major scaling and composition shortfalls of prior datasets. Empirical evaluation highlights the persistent limitations of current VLM/MLLM architectures in deep narrative reasoning. The PlotTree agent paradigm is validated as a structurally principled alternative, achieving strong empirical results on long-range story understanding even in the presence of knowledge-constrained or anonymized settings. The methodological and dataset innovations provide a robust testbed and reference paradigm for the next generation of video-language understanding systems.