NarrLV: Benchmark for Narrative Long Videos

• NarrLV is a benchmark designed to assess narrative richness in long video generation by formalizing narrative content through the Temporal Narrative Atom (TNA).
• It employs a controlled prompt generation pipeline and multi-level evaluation metrics, including fidelity, coverage, and coherence, to quantify narrative structure in generated videos.
• Empirical results indicate that while models maintain scene and object fidelity, they struggle with narrative unit coverage and coherence as the complexity increases.

NarrLV is a benchmark specifically designed for the comprehensive, narrative-centric evaluation of long video generation models. Developed in response to the rapid progress in foundation video generation technologies and the lack of narrative-oriented assessment metrics, NarrLV introduces a theoretically grounded framework inspired by film narratology. Its core contribution lies in the precise quantification of narrative richness and structure in generated videos via the operationalization of the Temporal Narrative Atom (TNA), an automatic pipeline for narrative prompt synthesis, and a multi-level evaluation paradigm using large vision-LLMs.

1. Theoretical Foundations and Definitions

NarrLV centers on the concept of the Temporal Narrative Atom (TNA), a formalization inspired by the “Beat” in film narratology. A TNA is defined as the smallest unit of narrative content that maintains continuous visual presentation in a video. Operationally, the introduction of a new TNA is triggered whenever a scene’s appearance, an object’s attribute, or an object’s action changes. The TNA count, denoted $n = |\{T_k\}_{k=1}^n|$, serves as a quantitative proxy for narrative richness. The parsing of textual prompts or generated video descriptions into TNAs is performed automatically, typically by instructing a LLM such as GPT-4o to segment the narrative into minimal continuous units.

Three dimensions of film-narrative change—scene attribute changes ($s_{att}$), object attribute changes ($t_{att}$), and object action changes ($t_{act}$)—are identified as the exclusive drivers of new TNA introduction. By design, temporal and spatial continuity are fixed, isolating these factors for controlled evaluation. Formally, the set of controlling factors is $F = [s_{att}, t_{att}, t_{act}]$ (Feng et al., 15 Jul 2025).

2. Prompt Generation Pipeline

The NarrLV benchmark is grounded in a controlled, large-scale prompt generation pipeline that systematically explores the narrative space. Scene–object pairs ($s_o$) are extracted from two large-scale datasets: VideoUFO-1M and DropletVideo-1M. LLMs (specifically, Qwen2.5-32B-Instruct) are employed to convert 200,000 human-written video prompts into a canonicalized JSON format {“scene”, “objects”}, yielding approximately 16,000 unique scene–object pairs.

For each factor $f \in F$ and each desired TNA count $n \in [1,6]$, 20 distinct scene–object pairs covering 14 broad scene categories are sampled. Prompt synthesis is handled by GPT-4o, which generates a structured text $p_{f,n}$ that includes an initial scene description and a stepwise sequence of $n$ atomic narrative units triggered by the chosen factor. The process ensures coverage across both the TNA count and the specific narrative change dimension. Post-processing yields a curated suite of 360 prompts, balanced across factors and narrative complexity (Feng et al., 15 Jul 2025).

3. Multi-Level Narrative-Expressive Evaluation Metric

NarrLV introduces a progressive evaluation metric that leverages multi-modal LLMs (MLLMs) for deep analysis. The evaluation suite consists of three scores, each probing distinct aspects of narrative alignment between a generated video $s_{att}$0 and a prompt $s_{att}$1:

• Narrative Element Fidelity ($s_{att}$2): Measures faithful rendering of the prompt’s basic elements (scene type, object attributes, layout). For each prompt, binary-judgment questions are generated via LLM and addressed by MLLM-based video analysis. $s_{att}$3 is computed as the mean accuracy over all questions.
• Narrative Unit Coverage ($s_{att}$4): Assesses whether each specified TNA in the prompt is present in the video. Coverage questions are posed for each TNA, with scores averaged over five MLLM responses per question.
• Narrative Unit Coherence ($s_{att}$5): Evaluates the coherence of transitions between consecutive TNAs. Transition questions are asked for each TNA boundary, combined with a robustness term measuring the proportion of existing TNAs ($s_{att}$6), to yield the final coherence score:

$s_{att}$7

These metrics are reported separately, though aggregation into a weighted sum is technically possible.

The evaluation protocol involves generating a video for each of the 360 prompts with target models, scoring them with the above metrics, and benchmarking alignment with human judgments via large-scale annotation (Feng et al., 15 Jul 2025).

4. Empirical Protocol and Model Coverage

NarrLV’s experimental suite evaluates both foundation models (e.g., Wan2.1-14B, CogVideoX1.5-5B, HunyuanVideo2.0, Open-Sora2.0) and long-video-specific architectures (e.g., TALC, FIFO-Diffusion, FreeNoise, FreeLong, RIFLEx) over resolutions up to 1280×720. The protocol mandates one video per prompt-model pair. Human judgments are collected across 1,800 comparisons of 600 video pairs, using prompts with n∈[2,6] TNAs, to measure the degree of alignment between the automatic scores and perceptual assessment.

NarrLV’s metrics show strong pairwise preference accuracy with human annotators: $s_{att}$8, $s_{att}$9, $t_{att}$0 for majority/unanimous conditions, significantly surpassing existing metrics such as VBench-2.0 (0.28–0.33) and StoryEval (0.41–0.56) (Feng et al., 15 Jul 2025).

5. Quantitative Findings and Capability Boundaries

Quantitative results reveal core characteristics and limitations of state-of-the-art video generation models:

• As TNA count $t_{att}$1 increases (from 1 to 6), $t_{att}$2 and $t_{att}$3 decline steeply, while $t_{att}$4 remains relatively stable (~0.7–0.8). This indicates that models maintain basic fidelity to scene and object details but fail to cover and coherently transition through a greater number of atomic narrative units.
• The effective number of TNAs expressed ($t_{att}$5) saturates around 2–3, even for n=6, across both foundation and long-video models.
• Object-attribute change prompts yield the highest fidelity but lowest coverage and coherence scores, pointing to difficulty in generating consistent attribute evolution over complex sequences.
• Foundation model choice exerts dominant influence over narrative expressiveness; long-form video modules yield only incremental improvements.

A practical implication is that reliable expression of narrative content is limited to short (n≤2) atomic sequences, and increased narrative complexity exacerbates model failure to cover or transition across all intended TNAs (Feng et al., 15 Jul 2025).

6. Limitations and Future Directions

Current models exhibit persistent constraints in long-form narrative expression. The inability to reliably render more than two TNAs per clip limits practical applicability to genuinely complex stories. While models accurately instantiate scene and object features for single events, their expressivity rapidly drops for more intricate temporal structures.

Future research directions include the development of improved foundation models with robust narrative compositionality, more effective long-form modules to sustain coherence across extended TNA chains, and the exploration of enhanced prompt construction and evaluation frameworks. NarrLV’s architecture additionally motivates work toward benchmarks that further increase the diversity and difficulty of narrative evaluation, ideally encompassing nested or hierarchical narrative structures (Feng et al., 15 Jul 2025).

7. Impact and Significance

NarrLV represents the first comprehensive benchmark tailored to the evaluation of narrative expressiveness in long video generation. Its rigorous theoretical foundation, systematic prompt generation pipeline, and MLLM-based metric provide a standardized methodology for diagnosing and quantifying the limitations of contemporary models. The benchmark’s superior alignment with human judgment and its ability to surface detailed capability boundaries position it as a central tool for the next phase of research in long-form video generation (Feng et al., 15 Jul 2025).