CineBench Benchmark: Video Compilation Evaluation

• CineBench is a benchmark for cinematic video compilation that translates free-form user instructions into coherent short videos while addressing challenges like contextual collapse and temporal fragmentation.
• It utilizes a diverse dataset of over 500 instruction–video pairs spanning multiple languages and genres, with detailed shot-level annotations to ensure narrative logic and editing precision.
• The framework incorporates robust metrics and baseline methods, enabling researchers to advance and compare multi-agent and sequence-to-sequence models for complex video editing tasks.

CineBench is the first comprehensive benchmark specifically designed for instruction-driven cinematic video compilation. Its primary objective is to enable and rigorously evaluate systems capable of translating free-form user instructions—such as “Show Andy Dufresne’s resourcefulness during the prison escape, in chronological order, with a 5-minute limit and a title card”—into coherent short-form videos compiled from long-form cinematic sources. By addressing challenges inherent to narrative-driven compilation, including contextual collapse and temporal fragmentation, CineBench offers a platform to advance automatic video editing methodologies beyond predefined tasks or shallow highlight extraction (Zhang et al., 12 Apr 2026).

1. Objectives, Scope, and Key Challenges

CineBench targets evaluation of systems handling instruction-driven compilation with high versatility. Unlike earlier approaches focusing on trailer generation or highlight detection, CineBench covers a broad range of creative intents, including varied narrative foci, complex editing operations, and adversarial consistency checks. Principal challenges encompass:

• Contextual Collapse: Compilation from thousands of overlapping shots often suffers semantic disconnects when naively operating at the keyword-retrieval level, leading to nonsensical or irrelevant clip selections.
• Temporal Fragmentation: Concern arises when relevant shots are concatenated without attention to story flow, impairing narrative logic and progression.
• Instruction Versatility: Task demands range from single- and multi-source compilation to execution of complex, non-linear, or adversarial instructions, together with standard editing operations such as text overlays, transitions, and music integration.

The benchmark thus operationalizes the evaluation of narrative understanding, cross-video reasoning, and editing skill (Zhang et al., 12 Apr 2026).

2. Dataset Composition and Annotation Protocol

CineBench is constructed from over 70 English- and Chinese-language films and television series spanning 1930–2020. This corpus includes genres such as action (30%), drama (25%), comedy (20%), romance (15%), and others (10%). The dataset comprises more than 500 instruction–video pairs, with source films containing between approximately 200 and 1,500 shots each.

Instructions are decomposed into four components for precise configuration:

1. Source Selection: Single or multi-source sampling.
2. Target Content: Specification of characters, events, or interleaved plots.
3. Temporal Requirements: Explicit ordering (chronological, non-linear, duration limits).
4. Editing Operations: Inclusion of titles, specific transition types, background music preferences, or cover images.

Ground-truth compilations are authored by five professional editors per video, with annotation spanning shot-level labels (ID, duration), event grouping, narrative abstractions (e.g., “climax”), and character profiling. Consistency is measured via inter-annotator agreement (Cohen’s Kappa of 0.61 on a 20% subset). Notably, 30% of instructions form an adversarial subset—factually impossible or inconsistent—to test a system’s refusal and detection capabilities. CineBench is released as a single unified set, with no predefined train/val/test splits, encouraging flexible experimental design (Zhang et al., 12 Apr 2026).

3. Task Formulation and Workflow

The central CineBench task is instruction-driven compilation. Formally:

• Input: Instruction $I$ (natural language), and source video set $V_{src}$.
• Output: Compiled script $L_{final}$ (ordered list: shot IDs and editing commands), and rendered video $V_{out}$.

Sub-tasks include:

1. Script Reverse-Engineering: Parsing $V_{src}$ into a hierarchical narrative memory $M$.
2. Shot Retrieval & Ordering: Selecting and sequencing shots in accordance with $I$.
3. Editing Execution: Performing operations such as transitions, overlays, and soundtrack insertion.

Submissions adopt a structured JSON schema. Optionally, rendered outputs may be submitted in mp4 format. This structure enables unambiguous benchmarking and model comparison.

4. Evaluation Metrics and Human Study Protocols

CineBench employs an extensive metric suite:

Metric	Definition	Range/Notes
Script-Video Consistency (SVC)	$$(1/N)\sum_{t=1}^N \cos(\text{embedding}(\hat{S}_t), \text{embedding}(V_{src_t}))$$	Shot-level cosine sim
Shot Coherence (SC)	$$(1/(N-1))\sum_{t=1}^{N-1} \text{VLM\_rate}(\hat{S}_t \to \hat{S}_{t+1})$$	[1,10]
Precision (P)	$|\hat{S} \cap G| / |\hat{S}|$
Recall (R)	$V_{src}$0
F1-score	$V_{src}$1
Temporal Correctness Score (TCS)	$V_{src}$2	$V_{src}$3
Narrative Logic (NL)	VLM scoring	[1,10]
Prompt Adherence (PA)	VLM scoring	[1,10]
Execution Success Rate (ESR)	Percentage of instructions completed without runtime errors
Adversarial Rejection Rate (ARR)	Percentage of negative instructions correctly refused
Compiled Quality (CQ)	Holistic VLM-based rating	[1,10]

Human evaluation involves four qualitative user studies on random samples (25 raters, 50 samples): Text-Prompt Alignment, Sequence Alignment, Narrative Coherence, and Overall Quality.

5. Baseline Methods and Comparative Results

Multiple systems are evaluated as baselines:

• Claude-3.7-sonnet (single VLLM)
• Gemini-2.5-pro (single VLLM)
• MetaGPT (multi-agent general planner)
• LAVE (casual video compilation)
• VideoAgent (all-in-one editing)
• CineAgents (“design-and-compose” multi-agent system)

Key quantitative results are as follows:

Method	SVC	SC	P (%)	R (%)	F1 (%)	TCS (%)	NL	PA	ESR (%)	ARR (%)	CQ
Claude	0.112	7.94	16.24	56.44	20.29	18.61	7.62	7.95	63.84	12.31	8.21
Gemini	0.131	8.57	30.88	76.21	41.59	37.41	8.08	8.47	74.11	23.85	8.67
MetaGPT	–	–	43.71	72.92	55.73	46.21	8.23	8.71	85.93	52.31	8.85
LAVE	–	–	38.22	63.21	44.60	35.37	8.60	8.43	70.34	40.28	8.53
VideoAgent	–	–	17.86	40.05	18.84	21.35	7.55	7.37	47.60	63.31	7.34
CineAgents	0.154	9.02	62.43	76.49	64.13	52.09	8.76	9.06	92.76	87.23	9.01

CineAgents achieves the highest performance across narrative grounding, retrieval precision, and overall reliability metrics. Multi-level contextual modeling and iterative narrative planning are identified as critical to mitigating contextual collapse and temporal fragmentation. Ablation studies confirm that omitting dialogue-character correspondence, hierarchical memory, or iterative planning results in metric deterioration (Zhang et al., 12 Apr 2026).

6. Usage, Extension, and Reproducibility

CineBench and CineAgents source code are available at https://github.com/tencent-ailab/CineBench. Installation requires Python 3.9+, PyTorch, HuggingFace Transformers, PaddleOCR, OpenCV, and Gemini API client. The installation procedure is as follows:

$V_{src}$4

Benchmark execution comprises:

1. Preprocessing source videos: $V_{src}$5
2. Compiling with a model: $V_{src}$6
3. Evaluation: $V_{src}$7

Extension involves adding new MP4 sources, updating metadata, and rerunning preprocessing. Integrating new models requires subclassing BaseCompiler, implementing generate_script() and assemble_video(), and registering the new method. Experimental splits can be defined in splits.yaml.

7. Research Significance and Outlook

With a corpus of 500+ instruction–video pairs, 11 quantitative metrics, and qualitative protocols, CineBench establishes a rigorous standard for cinematic video compilation research. It enables comparative evaluation across multiple axes, supporting future advances in narrative coherence, logic, and creative flexibility. A plausible implication is that CineBench’s fine-grained annotation and adversarial subsets provide an effective testbed for cross-modal reasoning and complex video editing tasks, stimulating progress in both multi-agent and sequence-to-sequence compilation methodologies (Zhang et al., 12 Apr 2026).