---
title: 'CineBench Benchmark: Video Compilation Evaluation'
url: https://www.emergentmind.com/topics/cinebench-benchmark
type: topic
---

# CineBench Benchmark: Video Compilation Evaluation

CineBench is the first comprehensive benchmark specifically designed for instruction-driven cinematic video compilation. Its primary objective is to enable and rigorously evaluate systems capable of translating free-form user instructions—such as “Show Andy Dufresne’s resourcefulness during the prison escape, in chronological order, with a 5-minute limit and a title card”—into coherent short-form videos compiled from long-form cinematic sources. By addressing challenges inherent to narrative-driven compilation, including contextual collapse and temporal fragmentation, CineBench offers a platform to advance automatic video editing methodologies beyond predefined tasks or shallow highlight extraction [2604.10456].

## 1. Objectives, Scope, and Key Challenges

CineBench targets evaluation of systems handling instruction-driven compilation with high versatility. Unlike earlier approaches focusing on trailer generation or highlight detection, CineBench covers a broad range of creative intents, including varied narrative foci, complex editing operations, and adversarial consistency checks. Principal challenges encompass:

- **Contextual Collapse**: Compilation from thousands of overlapping shots often suffers semantic disconnects when naively operating at the keyword-retrieval level, leading to nonsensical or irrelevant clip selections.
- **Temporal Fragmentation**: Concern arises when relevant shots are concatenated without attention to story flow, impairing narrative logic and progression.
- **Instruction Versatility**: Task demands range from single- and multi-source compilation to execution of complex, non-linear, or adversarial instructions, together with standard editing operations such as text overlays, transitions, and music integration.

The benchmark thus operationalizes the evaluation of narrative understanding, cross-video reasoning, and editing skill [2604.10456].

## 2. Dataset Composition and Annotation Protocol

CineBench is constructed from over 70 English- and Chinese-language films and television series spanning 1930–2020. This corpus includes genres such as action (30%), drama (25%), comedy (20%), romance (15%), and others (10%). The dataset comprises more than 500 instruction–video pairs, with source films containing between approximately 200 and 1,500 shots each.

Instructions are decomposed into four components for precise configuration:
1. **Source Selection**: Single or multi-source sampling.
2. **Target Content**: Specification of characters, events, or interleaved plots.
3. **Temporal Requirements**: Explicit ordering (chronological, non-linear, duration limits).
4. **Editing Operations**: Inclusion of titles, specific transition types, background music preferences, or cover images.

Ground-truth compilations are authored by five professional editors per video, with annotation spanning shot-level labels (ID, duration), event grouping, narrative abstractions (e.g., “climax”), and character profiling. Consistency is measured via inter-annotator agreement (Cohen’s Kappa of 0.61 on a 20% subset). Notably, 30% of instructions form an adversarial subset—factually impossible or inconsistent—to test a system’s refusal and detection capabilities. CineBench is released as a single unified set, with no predefined train/val/test splits, encouraging flexible experimental design [2604.10456].

## 3. Task Formulation and Workflow

The central CineBench task is instruction-driven compilation. Formally:

- **Input**: Instruction $I$ (natural language), and source video set $V_{src}$.
- **Output**: Compiled script $L_{final}$ (ordered list: shot IDs and editing commands), and rendered video $V_{out}$.

Sub-tasks include:
1. **Script Reverse-Engineering**: Parsing $V_{src}$ into a hierarchical narrative memory $M$.
2. **Shot Retrieval & Ordering**: Selecting and sequencing shots in accordance with $I$.
3. **Editing Execution**: Performing operations such as transitions, overlays, and soundtrack insertion.

Submissions adopt a structured JSON schema. Optionally, rendered outputs may be submitted in mp4 format. This structure enables unambiguous benchmarking and model comparison.

## 4. Evaluation Metrics and Human Study Protocols

CineBench employs an extensive metric suite:

| Metric                 | Definition                                              | Range/Notes               |
|------------------------|--------------------------------------------------------|---------------------------|
| Script-Video Consistency (SVC) | $(1/N)\sum_{t=1}^N \cos(\text{embedding}(\hat{S}_t), \text{embedding}(V_{src_t}))$ | Shot-level cosine sim     |
| Shot Coherence (SC)    | $(1/(N-1))\sum_{t=1}^{N-1} \text{VLM\_rate}(\hat{S}_t \to \hat{S}_{t+1})$ | [1,10]                    |
| Precision (P)          | $|\hat{S} \cap G| / |\hat{S}|$                         |                          |
| Recall (R)             | $|\hat{S} \cap G| / |G|$                               |                          |
| F1-score               | $2PR/(P+R)$                                             |                          |
| Temporal Correctness Score (TCS) | $(\sum_{i \in \text{CorrectOrder}} d_i)/ (\sum_{i=1}^N d_i)$ | $[0,1]$                  |
| Narrative Logic (NL)   | VLM scoring                                            | [1,10]                   |
| Prompt Adherence (PA)  | VLM scoring                                            | [1,10]                   |
| Execution Success Rate (ESR) | Percentage of instructions completed without runtime errors |                   |
| Adversarial Rejection Rate (ARR) | Percentage of negative instructions correctly refused |         |
| Compiled Quality (CQ)  | Holistic VLM-based rating                              | [1,10]                   |

Human evaluation involves four qualitative user studies on random samples (25 raters, 50 samples): Text-Prompt Alignment, Sequence Alignment, Narrative Coherence, and Overall Quality.

## 5. Baseline Methods and Comparative Results

Multiple systems are evaluated as baselines:

- Claude-3.7-sonnet (single VLLM)
- Gemini-2.5-pro (single VLLM)
- MetaGPT (multi-agent general planner)
- LAVE (casual video compilation)
- VideoAgent (all-in-one editing)
- CineAgents (“design-and-compose” multi-agent system)

Key quantitative results are as follows:

| Method      | SVC   | SC   | P (%) | R (%) | F1 (%) | TCS (%) | NL   | PA   | ESR (%) | ARR (%) | CQ   |
|-------------|-------|------|-------|-------|--------|---------|------|------|---------|---------|------|
| Claude      | 0.112 | 7.94 | 16.24 | 56.44 | 20.29  | 18.61   | 7.62 | 7.95 | 63.84   | 12.31   | 8.21 |
| Gemini      | 0.131 | 8.57 | 30.88 | 76.21 | 41.59  | 37.41   | 8.08 | 8.47 | 74.11   | 23.85   | 8.67 |
| MetaGPT     |   –   |   –  | 43.71 | 72.92 | 55.73  | 46.21   | 8.23 | 8.71 | 85.93   | 52.31   | 8.85 |
| LAVE        |   –   |   –  | 38.22 | 63.21 | 44.60  | 35.37   | 8.60 | 8.43 | 70.34   | 40.28   | 8.53 |
| VideoAgent  |   –   |   –  | 17.86 | 40.05 | 18.84  | 21.35   | 7.55 | 7.37 | 47.60   | 63.31   | 7.34 |
| CineAgents  | 0.154 | 9.02 | 62.43 | 76.49 | 64.13  | 52.09   | 8.76 | 9.06 | 92.76   | 87.23   | 9.01 |

CineAgents achieves the highest performance across narrative grounding, retrieval precision, and overall reliability metrics. Multi-level contextual modeling and iterative narrative planning are identified as critical to mitigating contextual collapse and temporal fragmentation. Ablation studies confirm that omitting dialogue-character correspondence, hierarchical memory, or iterative planning results in metric deterioration [2604.10456].

## 6. Usage, Extension, and Reproducibility

CineBench and CineAgents source code are available at https://github.com/tencent-ailab/CineBench. Installation requires Python 3.9+, PyTorch, HuggingFace Transformers, PaddleOCR, OpenCV, and Gemini API client. The installation procedure is as follows:

```bash
git clone https://github.com/tencent-ailab/CineBench.git
cd CineBench
pip install -r requirements.txt
```

Benchmark execution comprises:
1. Preprocessing source videos:
   ```bash
   python preprocess_videos.py --input_dir path/to/movies --output_dir narrative_memory/
   ```
2. Compiling with a model:
   ```bash
   python compile.py --instruction_file instructions.json --memory_dir narrative_memory/ --out_dir outputs/
   ```
3. Evaluation:
   ```bash
   python evaluate.py --predictions outputs/ --ground_truth gt_scripts/ --metrics all
   ```

Extension involves adding new MP4 sources, updating metadata, and rerunning preprocessing. Integrating new models requires subclassing `BaseCompiler`, implementing `generate_script()` and `assemble_video()`, and registering the new method. Experimental splits can be defined in `splits.yaml`.

## 7. Research Significance and Outlook

With a corpus of 500+ instruction–video pairs, 11 quantitative metrics, and qualitative protocols, CineBench establishes a rigorous standard for cinematic video compilation research. It enables comparative evaluation across multiple axes, supporting future advances in narrative coherence, logic, and creative flexibility. A plausible implication is that CineBench’s fine-grained annotation and adversarial subsets provide an effective testbed for cross-modal reasoning and complex video editing tasks, stimulating progress in both multi-agent and sequence-to-sequence compilation methodologies [2604.10456].

Source: https://www.emergentmind.com/topics/cinebench-benchmark