- The paper introduces a novel benchmark that leverages 1,005 full-length movies with dense multimodal narrative event annotations to assess long video understanding.
- It employs an LLM-assisted annotation protocol using PropBank roles and defines tasks like ETD, EL, EAE, and ERE evaluated at scene-level granularity.
- Experimental results reveal significant challenges in narrative abstraction and non-linear temporal reasoning, with current models showing limited progress.
NEST: Narrative Event Structures in Time for Long Video Understanding
Motivation and Context
NEST ("Narrative Event Structures in Time for Long Video Understanding" (2606.19706)) addresses critical limitations of current vision-LLMs (VLMs) and benchmarks in the domain of long-form video understanding. Existing approaches are frequently restricted to retrieval or atomic action recognition over short clips, with minimal treatment of narrative abstraction, event composition, or long-range dependencies. This gap precludes comprehensive evaluation of models' ability to reason about hierarchical event structure, causal links, and cross-temporal relations—capabilities essential for real-world video understanding in domains such as education, entertainment, and automated video analysis.
Dataset Design and Annotation Protocol
NEST comprises 1,005 full-length movies (average duration ~98 minutes), spanning diverse genres and sourced from open-domain repositories and public databases. Each movie is densely annotated with approximately 102 multimodal narrative events grounded in visual, dialogue, and audio modalities. The event ontology is constrained by PropBank semantic roles to ensure structured definition and consistent argument extraction. Relations between events include temporal (before, after, overlap), causal, hierarchical, preconditioned, and coreference links, supporting high-fidelity modeling of narrative structure across extensive temporal gaps (average event pairs separated by ~32 minutes).
Annotation integrates LLMs for initial extraction and consistency checking, validated via a two-step protocol combining audio description (local visual grounding) and plot/script cross-referencing (global narrative consistency). The annotation pipeline yields a large-scale SILVER dataset (LLM-assisted) and a human-annotated GOLD benchmark, with inter-annotator weighted Cohen's K reaching ~0.57 and GOLD-SILVER agreement ~0.50—indicating that the pipeline achieves 86-88% of human-level annotation consistency.
Benchmark Tasks and Evaluation Methodology
NEST defines four fundamental tasks for narrative event understanding:
- Event Trigger Detection (ETD): Identify narrative event triggers from multimodal evidence.
- Event Localization (EL): Determine temporal boundaries of events within the movie.
- Event Argument Extraction (EAE): Extract structured arguments (participants, objects, locations) for each event.
- Event Relation Extraction (ERE): Classify relations between event pairs (temporal, causal, hierarchical, preconditioned, coreference, no_relation).
Each task is evaluated on scene-level granularity, reflecting the inherent subjectivity of narrative event boundaries and the limitations of current temporal grounding methods. ETD and EAE utilize an LLM-as-a-judge system for semantic equivalence evaluation; ERE employs closed-set macro-F1 scoring. The benchmark supports both multimodal and text-only variant evaluations.
Experimental Results
Baseline evaluations include state-of-the-art multimodal and long-context models: Qwen3-VL, Qwen3-Omni, Qwen2.5-VL, InternVL3.5, Video-LLaMA3, LongVU, OVIS2.5, Flash-VStream-Qwen, and others. The core findings are as follows:
- Narrative Event Discovery Remains Largely Unsolved: ETD (<8%), EL (<6%), and EAE (<11%) accuracy across all models, despite synonym-tolerant evaluation protocols. The bottleneck is narrative abstraction and temporal grounding, not visual coverage or frame sampling density.
- Event Relation Extraction is More Tractable Conditioned on Event Discovery: ERE achieves 35.45% macro-F1 zero-shot (Qwen2.5-VL 32B) and 44.42% macro-F1 after LoRA fine-tuning, but performance varies sharply by relation type. Coreference and causal relations are learned more reliably; temporal, preconditioned, and hierarchical connections pose persistent challenges, especially when events are separated by large temporal intervals.
- Failure Modes are Distinct: ETD errors are dominated by wrong narrative event and atomic verb defaults; EAE errors are primarily due to entity confusions, reflecting inadequate long-range entity tracking; EL failures are overwhelmingly wrong scene localization (98.6%), indicating models are confidently wrong in selecting event boundaries; ERE errors are mostly over-prediction of NO_RELATION, revealing the difficulty in inferring narrative links across distant events.
- Non-Linear Temporal Reasoning is Catastrophically Difficult: Flashback relations induce near-zero F1 for most models, with implicit linear timeline assumptions limiting robustness against common narrative constructs.
- Fine-Tuning Benefits Conditional Reasoning, Not Event Discovery: Finetuned models show significant improvements in ERE (from 35.45% to 44.42% F1) but only marginal gains in ETD and EL, suggesting that robust narrative abstraction and event grounding require advances beyond SFT on task-specific data.
Implications and Theoretical Impact
NEST establishes a rigorous benchmark for narrative event extraction and composition in long-form multimodal video, surfacing the deficiencies of current VLMs in narrative abstraction, hierarchical event structuring, and temporal reasoning. The dataset's scale, annotation quality, and event ontology enable systematic investigation of grounded event discovery, long-range entity tracking, and complex cross-modal inference—areas that are not adequately covered by prior benchmarks (which are limited by short duration, multiple-choice biases, or retrieval focus).
The results indicate that advances in long-context modeling—such as increased token streams and memory persistence—do not automatically yield improvements in narrative understanding. Instead, explicit mechanisms for narrative abstraction, symbolic event graphs, temporally-aware retrieval augmentation, and hybrid neuro-symbolic reasoning are central to bridging the gap between human and model performance.
Limitations and Future Directions
Annotation relies on LLMs for verification and judge evaluation, introducing potential biases and alignment artifacts. Scene-level evaluation is a pragmatic proxy for temporal grounding but may not capture fine-grained narrative boundaries. Cross-scene composite events (e.g., gradual character transformations or subplots across non-adjacent scenes) are not represented, motivating further elaboration of annotation and evaluation protocols.
Future research should prioritize:
- Development of narrative abstraction modules integrating chain-of-thought over prior multimodal context.
- Long-range entity tracking using visual re-ID and dialogue-based coreference mechanisms.
- Enhancement of non-linear temporal reasoning capacities.
- Retrieval-augmented or memory-aware grounding methods to scale reasoning over hours-long input.
- Integration of symbolic reasoning structures (event graphs, temporal logic) with neural representations.
Conclusion
NEST provides a comprehensive, scalable benchmark for narrative event understanding in full-length movies, demonstrating that current vision-LLMs fail at grounded event discovery and hierarchical reasoning in long-form video. The dataset, annotation pipeline, and results underscore the necessity of new architectural and algorithmic advances for genuine narrative-level comprehension and temporal abstraction in multimodal AI.