LoCoT2V-Bench: Long-Form Text-to-Video Eval

• LoCoT2V-Bench is a benchmark evaluating long-form, complex text-to-video generation using real YouTube-derived prompts that encode narrative and event dynamics.
• It employs a multi-dimensional evaluation framework measuring static quality, text-video alignment, temporal quality, content clarity, and HERD to capture both low-level and high-level narrative aspects.
• Analysis across nine models shows strong low-level performance but significant challenges in maintaining inter-event consistency, narrative flow, and thematic adherence.

Searching arXiv for the specified benchmark paper and closely related benchmarks to ground the article. LoCoT2V-Bench is a benchmark for long-form and complex text-to-video generation, designed to evaluate long video generation under complex input conditions rather than short clips with simplified prompts. It uses 240 prompts derived from real-world YouTube videos across 18 themes, with prompts that are long, realistic, and explicitly structured around scene transitions, camera motion, event dynamics, multi-actor interactions, and thematic or emotional intent. Its central contribution is a multi-dimensional evaluation framework spanning static quality, text-video alignment, temporal quality, content clarity, and Human Expectation Realization Degree (HERD), with particular emphasis on event-level adherence, long-range temporal dependencies, and abstract dimensions such as narrative flow, emotional response, and character development. In the reported evaluation of nine representative long video generation models, low-level visual and temporal performance is generally strong, while inter-event consistency, fine-grained alignment, and high-level thematic or narrative adherence remain weak (Zheng et al., 30 Oct 2025).

1. Scope, motivation, and benchmark position

LoCoT2V-Bench is motivated by a specific gap in long video generation evaluation: existing benchmarks mostly target short clips and simplified prompts, and they emphasize low-level metrics such as visual fidelity and motion smoothness while overlooking event-level adherence, multi-event temporal dependencies, and abstract, human-facing dimensions such as narrative flow, themes, and emotional impact (Zheng et al., 30 Oct 2025). Existing long-video efforts are described as either keeping prompts simple, which limits realism, or providing only a small number of complex scripts, which restricts generalization and robust evaluation.

The benchmark is therefore defined around long-form, complex text-to-video generation. Its design is grounded in real-world videos rather than synthetic prompt construction alone, and its prompts encode scene changes, event progression, camera operations, and higher-level narrative expectations. This gives it a dual role: it is both an evaluation suite for long video generation and a diagnostic framework for separating coarse visual competence from failures in multi-event planning and narrative realization.

Relative to related benchmarks, LoCoT2V-Bench is characterized by a stronger long-form and prompt-complexity orientation. Compared to EvalCrafter, VBench, VBench++ (long), and VBench 2.0 complex suites, it is described as providing more and longer complex prompts, with an average prompt length of 236.66 words and average complexity of 8.75/10, together with explicit event structures, camera motions, and transitions. It also introduces HERD and Content Clarity, extending evaluation beyond low-level fidelity and motion into narrative coherence, themes, emotion, and character development (Zheng et al., 30 Oct 2025).

2. Benchmark composition and prompt construction

The benchmark is built from real-world videos collected from YouTube using yt-dlp. The initial collection consists of thousands of 30–60 s short-form videos, after which the final set is reduced to 240 videos, evenly distributed across 18 themes. For analysis, those themes are pooled into three broad categories: Human Real Life, Nature Exploration, and Virtual Entertainment (Zheng et al., 30 Oct 2025). The collected real videos are 30–60 s in duration, but generation length is not fixed by the benchmark and depends on model pipelines.

LoCoT2V-Bench is evaluation-only. It uses a single evaluation suite of 240 prompts and has no train/test split. Prompts and evaluation are in English. The source videos often contain multiple shots or scenes with explicit transitions, and the prompts encode camera operations such as pan, zoom, and tracking, as well as settings, subjects, actions, multi-actor interactions, and narrative constraints.

Prompt construction follows three stages. First, videos are collected and manually filtered for quality and relevance. Second, prompts are generated from video content using strong video-capable MLLMs, including the Qwen2.5-VL family, then refined iteratively through self-refine and manually corrected for factual errors. Third, higher-level assessment information is inferred from the original videos with Seed1.5-VL and fused into prompts so that emotional response, narrative flow, character development, visual style, themes, interpretive depth, and overall impression are encoded as explicit expectations (Zheng et al., 30 Oct 2025).

Prompt complexity is one of the benchmark’s defining properties. The prompts have an average length of 236.66 words, are mostly 200–300 words long, and score highly on semantic, structural, and control complexity, with average complexity 8.75/10 as scored by DeepSeek-V3.1. The complex elements explicitly include scene transitions, multi-event progressions with temporal order, camera motions, fine-grained actions, spatial relations, lighting and visual style constraints, and thematic or emotional arcs.

3. Event schema, prompt taxonomy, and annotation design

The benchmark’s prompt design is organized around an event schema in which each event comprises the fields {event description, subject, setting, action, camera motion}. This schema serves both prompt construction and evaluation. In practical terms, it defines the units against which event-level alignment and temporal consistency are assessed, and it formalizes what counts as a correctly realized event in long-form video generation (Zheng et al., 30 Oct 2025).

Prompts encode not only local event content but also temporal dependencies and narrative constraints. These include inter-event ordering, continuity across scenes, camera-driven transitions, and higher-level narrative or thematic requirements such as an atmosphere evolving from calm to tense or relationships developing across scenes. The benchmark further integrates dimension-specific expectations as binary-evaluable cues for HERD, so that abstract aspects of the source video become operationalized in downstream evaluation.

The annotation and prompting templates are tool-oriented rather than human-annotation-oriented. The reported templates guide overall video description generation for alignment, event extraction from both text and generated video descriptions, human action detection and smoothness queries, content clarity scoring on a 0–4 scale with rationale, and HERD question generation with polarity annotation. This suggests that the benchmark is structured as a tightly coupled prompt-and-evaluation system in which MLLMs and VLMs are responsible for much of the semantic decomposition and judgment pipeline, rather than serving only as auxiliary analyzers.

4. Multi-dimensional evaluation framework

The evaluation framework comprises five dimensions and 26 sub-dimensions. It is designed to separate low-level visual quality from alignment, temporal continuity, communicability, and abstract expectation realization (Zheng et al., 30 Oct 2025).

Dimension	Main components	Core tools or procedures
Static Quality (SQ)	Aesthetic Quality, Technical Quality	Aesthetic Predictor V2.5, DOVER++
Text-Video Alignment (TVA)	Overall Alignment, Event-level Alignment	Qwen2.5-VL-7B, DeepSeek-V3.1, Hungarian algorithm
Temporal Quality (TQ)	Dynamic Degree, Motion Smoothness, Human Action, Temporal Flickering, Transition Smoothness, Warping Error, Semantic Consistency, intra-/inter-event consistency	VBench metrics, EvalCrafter metrics, PySceneDetect, RAFT, Time-R1-7B, Grounded-SAM-2
Content Clarity (CC)	Theme Clarity, Logical Structure, Information Completeness, Information Consistency	MLLM scoring with multiple randomized trials
HERD	Emotional Response, Narrative Flow, Character Development, Visual Style, Themes, Interpretive Depth, Overall Impression	Seed1.5-VL, DeepSeek-V3.1, MLLM VQA

Within Static Quality, Aesthetic Quality uses Aesthetic Predictor V2.5 on 1 fps sampled frames and normalizes scores using a Relative Reference Upper Bound derived from a high-quality image set, specifically Text-to-Image-2M, data_1024_10K, with the mean of the top 10% scores as upper bound. Technical Quality uses DOVER++, with long videos segmented into clips of at most 10 s and scores averaged.

Text-Video Alignment has two levels. Overall Alignment is computed by generating an overall textual description of the generated video with Qwen2.5-VL-7B and comparing it semantically with the prompt base, excluding evaluation add-ons. Event-level Alignment is finer-grained. Ground-truth events are extracted from the prompt base with DeepSeek-V3.1, events are extracted from the generated video description as well, and maximum-weight bipartite matching is performed with the Hungarian algorithm based on semantic similarity of event descriptions. The paper defines the event score as

$$S_{\text{event}}= \left(1-\frac{I}{I_{\max}}\right)\cdot \frac{1}{N}\sum_{i=1}^{N} \left[ S_{\text{sem}}^{(i)}\cdot \frac{1}{4}\sum_{f\in\{\text{subj,set,act,cam}\}} S_f^{(i)} \right].$$

Here, $I$ is the number of inversions in matched event order, $I_{\max}$ is the maximum possible inversions, and $N$ is the number of matched pairs. The formulation makes temporal mis-ordering part of alignment rather than treating it as a separate defect.

Temporal Quality combines adopted and newly introduced metrics. Dynamic Degree and Motion Smoothness are adopted from VBench. Warping Error and Semantic Consistency are taken from EvalCrafter. Human Action is evaluated through a two-step MLLM process in which actions are first extracted from the prompt and then checked for occurrence and smoothness through yes/no questions on the generated video. Transition Smoothness is a novel metric for multi-scene transitions: scene cuts are detected with PySceneDetect, a similarity sequence is computed from normalized pixel MAE, SSIM, SigLIP feature similarity, and motion consistency from RAFT, abruptness is quantified by normalized variance, and smoothness is defined as $1-$ abruptness. The appendix gives the corresponding equations:

$$s_j=\alpha_1 \hat s_{\text{mae}}+\alpha_2 \hat s_{\text{ssim}}+\alpha_3 \hat s_{\text{siglip}}+\alpha_4 \hat s_{\text{mc}}, \qquad \sum \alpha_k = 1,$$

$$A(t_i)=\frac{\mathrm{Var}(S(t_i))\cdot b}{\mathrm{Var}(S(t_i))\cdot b+c}, \qquad S(t_i)=1-A(t_i),$$

and

$$\mathrm{Score}_{ts}=\frac{1}{N_1}\sum_{i=1}^{N_1} S(t_i).$$

The benchmark also measures intra-/inter-event temporal consistency for subjects and backgrounds. Events are segmented with Time-R1-7B, subjects are segmented with Grounded-SAM-2, subject crops and background regions are formed by masking, and framewise features are compared with cosine similarity. The distinction between intra-event and inter-event consistency is crucial because it separates short-range stability within an event from long-range identity and background persistence across events.

Content Clarity is defined as whether a video communicates a coherent, comprehensible narrative, emphasizing semantic coherence and viewer comprehension. It uses Theme Clarity, Logical Structure, Information Completeness, and Information Consistency, each scored on a 0–4 scale by an MLLM with multiple randomized trials, normalized by 4, and averaged. The formal definition is

$$S(v)=\frac{1}{D}\sum_{d=1}^{D} s_d(v), \qquad s_d(v)=\frac{1}{R}\sum_{i=1}^{R} \frac{s_d^{(i)}(v)}{4}, \qquad D=4.$$

HERD is the benchmark’s high-level human-centric metric. It is polarity-aware and question-based, covering Emotional Response, Narrative Flow, Character Development, Visual Style, Themes, Interpretive Depth, and Overall Impression. For each prompt, multi-dimensional assessments are produced from the original real-world video with Seed1.5-VL, converted by DeepSeek-V3.1 into multiple yes/no questions per dimension, and labeled with positive or negative polarity. Answers are yes, no, or unclear, and unclear is excluded. The score is

$$S_{\text{HERD}}= \frac{\sum_{q\in Q} I(q)} {\sum_{q\in Q} \mathbf{1}[A(q)\in\{\text{yes},\text{no}\}]},$$

where $I$0 if the answer matches the polarity condition and $I$1 otherwise. A common misconception is that HERD is a direct human-rating protocol; in the benchmark as reported, it is an MLLM-driven evaluation reflecting human expectations inferred from original videos rather than inter-rater judgments from human annotators.

5. Evaluated models and empirical findings

The benchmark evaluates nine representative open-source long video generation methods spanning diverse paradigms: FreeNoise, MEVG, FreeLong, FIFO-Diffusion, DiTCtrl, CausVid, SkyReels-V2 (540p), Vlogger, and VGoT (Zheng et al., 30 Oct 2025). These methods include tuning-free long video diffusion, multi-event generation with structure-guided sampling, spectral temporal attention, infinite generation with FIFO frame queues, MM-DiT attention control, autoregressive diffusion distilled from a bidirectional teacher, multi-stage training with motion-specific RL and diffusion forcing, an LLM-directed vlog pipeline, and a modular “VideoGen-of-Thought” system with keyframes and cross-shot smoothing.

Category	Best	Second
Static Quality	VGoT, 91.15%	CausVid, 75.99%
Text-Video Alignment	CausVid, 66.47%	DiTCtrl, 63.06%
Temporal Quality	SkyReels-V2, 79.49%	FIFO-Diffusion, 75.58%
Content Clarity	FIFO-Diffusion, 80.82%	VGoT, 79.79%
HERD	VGoT, 63.74%	CausVid, 63.55%
Overall average	VGoT, 72.17%	SkyReels-V2, 69.64%

The aggregate pattern is that no single system dominates all aspects. VGoT leads Static Quality, HERD, and overall average; CausVid leads Text-Video Alignment; SkyReels-V2 leads Temporal Quality; and FIFO-Diffusion leads Content Clarity. This distribution suggests that long-form generation remains decomposable into partially independent competencies rather than collapsing into a single dominant notion of quality.

Sub-dimension results reinforce that interpretation. SkyReels-V2 is best on Dynamic Degree and Transition Smoothness. MEVG is best on Motion Smoothness, Warping Error, and Temporal Flickering. VGoT is best on Semantic Consistency, Emotional Response, and Character Development. Vlogger is best on Visual Style. CausVid is best on Human Action, Themes, Interpretive Depth, and Overall Impression. FIFO-Diffusion is strongest for inter-event consistency in both subject and background, indicating superior long-range identity and background stability across events (Zheng et al., 30 Oct 2025).

The benchmark’s principal empirical finding is that models excel in static quality and short-term temporal stability but exhibit major gaps in fine-grained event realization and long-range narrative continuity. Event-level alignment is markedly lower than overall alignment, indicating difficulty in following prompt-specified subjects, actions, and camera motions at event resolution. Long-range temporal issues appear in low inter-event subject and background consistency and only moderate transition smoothness. HERD results show that high-level thematic and narrative adherence remains especially difficult, particularly for narrative flow, interpretive depth, and character development.

The paper also reports low correlations between some dimensions. Static quality has limited linear association with most other dimensions, with the strongest correlation being with Content Clarity at Pearson $I$2. Event-level alignment versus event-level temporal consistency shows low correlation, with Pearson $I$3–$I$4 depending on the pair, indicating minimal metric entanglement at the sample level. This suggests that a video can be temporally consistent for extracted subjects while still failing to follow the prompt’s event structure.

Failure modes are correspondingly specific: mis-ordered or missing events, poor adherence to camera instructions, drift in identities or backgrounds across shots, abrupt transitions, narratives that do not progress or develop characters, and limited ability to realize abstract themes or evoke intended emotions.

6. Sensitivity, limitations, usage, and significance

The benchmark includes prompt-complexity and theme-based analyses. Higher semantic and structural complexity degrades performance across multiple dimensions, indicating that models struggle to parse and execute complex semantics and structures. Control complexity has less pronounced effects, which the paper attributes to the fact that most baselines already underperform on fine-grained control requirements such as style, camera, and dynamics. Static quality and coarse text-video alignment are relatively less sensitive to complexity than the other dimensions. Theme category robustness is reported as broadly similar across Human Real Life, Nature Exploration, and Virtual Entertainment, with only minor differences and slightly higher alignment for Nature Exploration, which tends to have shorter prompts of intermediate complexity (Zheng et al., 30 Oct 2025).

Several limitations are explicitly identified. The benchmark is English-only and contains 240 samples, which provides broad thematic coverage but remains modest in scale. Because prompts are derived from real videos of 30–60 s, the source distribution may bias the benchmark toward short-form storytelling structures rather than movie-like narratives. MLLM-driven judgments in Content Clarity, HERD, and action verification can inherit model biases, and no human inter-rater reliability is reported. Event extraction and segmentation also depend on LLM or VLM outputs.

The benchmark is designed for practical reuse. Prompt data, evaluation code, and results are to be released at https://anonymous.4open.science/r/LoCoT2V-Bench-1518/; an initial JSON prompt suite is already available, and the full code is to be released after reorganization and verification. Running the benchmark on a new model consists of generating videos for the 240 prompts and applying the provided evaluation scripts to compute the five-dimension suite. Dependencies include Aesthetic Predictor V2.5, DOVER++, PySceneDetect, RAFT, SigLIP features, Time-R1-7B, Grounded-SAM-2, Qwen2.5-VL in 7B and 72B variants, DeepSeek-V3.1, and Seed1.5-VL. Videos are collected under YouTube ToS, and the benchmark reports strict filtering and safety guidelines, with no PII or harmful content.

The broader significance of LoCoT2V-Bench lies in the type of progress it makes measurable. It enables event- and narrative-aware evaluation of long-form, complex text-to-video generation across five major dimensions and 26 sub-dimensions, including novel event-level and high-level human-centric criteria. A plausible implication is that future improvements in long video generation will need to focus less on isolated frame quality and more on planning, memory, cross-event continuity, camera semantics, and abstract narrative realization. In that sense, LoCoT2V-Bench functions not only as a benchmark but also as a specification of what long-form text-to-video generation is still failing to achieve at scale (Zheng et al., 30 Oct 2025).