CutBench: Cinematic Next-Shot Benchmark

• CutBench is a benchmark suite designed to evaluate next-shot generation models by simulating cinematic editing transitions with extensive hierarchical annotations.
• It features a rigorously curated dataset of professionally edited shot pairs covering four canonical cut patterns to ensure balanced narrative and visual continuity.
• Its evaluation protocols combine automatic metrics such as DINO, CLIP, and FID with human perceptual studies to assess cinematic authenticity and editing adherence.

CutBench is a benchmark suite for the evaluation of next-shot generation models, specifically targeting the modeling of professional editing patterns and strict cinematic continuity in multi-shot filmmaking workflows. Designed to bridge the gap between purely visual consistency and narrative-driven video editing, CutBench is constructed from a rigorously curated dataset and is coupled with hierarchical prompt-based annotations and multi-faceted evaluation protocols. It serves both as a benchmark and as an annotation methodology for comprehensive assessment of generative models in the Next Shot Generation (NSG) task (He et al., 11 Aug 2025).

1. Dataset Origin and Construction

CutBench is derived from CuratedCuts, a high-quality subset of the larger RawCuts collection. RawCuts consists of approximately 200,000 adjacent shot-pairs sourced to cover a broad range of professional cinematic material. CuratedCuts is distilled from RawCuts via human oversight, resulting in “thousands” of professionally edited shot-pairs—each pair demonstrating robust narrative and visual continuity conforming to canonical editing styles. From CuratedCuts, CutBench reserves “hundreds of diverse movie shot images” for evaluation purposes, ensuring no overlap with training data.

Shot pairs in CutBench systematically represent four canonical cut patterns central to film editing:

• Shot/Reverse Shot (primarily for dialogue sequences)
• Cut-In / Cut-Out (emphasizing detail versus wider context)
• Cutaway (providing external or subjective context)
• Multi-Angle (offering varied viewpoints of a single subject)

This careful curation ensures that the benchmark covers a spectrum of shot transitions emblematic of practical cinematic narratives (He et al., 11 Aug 2025).

2. Annotation Methodology: Hierarchical Prompting

Annotation in CutBench uses a hierarchical structure, reflecting both the relational context between shots and per-shot cinematic attributes. The automated annotation pipeline employs Gemini-2.0-flash to generate two levels of prompts for every shot pair $(S_{cond}, S_{tgt})$:

• Relational Prompt ($P^{rel}$): Encodes inter-shot editing style, shared scene and character context, and spatial-temporal continuity.
• Individual Prompts ($P^{ind}_{cond}, P^{ind}_{tgt}$): Capture content elements—subjects, expressions, costumes, backgrounds—and structured cinematographic details such as framing, camera angle, and focal length.

All automated annotations undergo light human verification, and, during model training, detailed components of $P^{ind}$ are subjected to a 20% random dropout. This dropout simulates real-world prompt incompleteness, promoting model robustness to annotation variability (He et al., 11 Aug 2025).

3. Data Splitting and Statistical Properties

CutBench divides CuratedCuts into train, validation, and test splits, reserving several hundred pairs for the test set. The overall editing-pattern distribution is designed to be approximately balanced: each of the four canonical cut categories constitutes roughly 20–30% of the test set, thereby preventing dominance by any single transition pattern.

Quantitative statistics are as follows:

Subset	Approximate Size	Source	Editing Pattern Distribution
RawCuts	200,000 pairs	Extracted	Unbalanced, diverse
CuratedCuts	"Thousands"	Filtered	Professional, balanced
CutBench Test	"Hundreds"	Held-out	~20–30% per cut type

A specific sample of 100 test cases was selected at random for the primary human study (He et al., 11 Aug 2025).

4. Evaluation Protocols and Metrics

Evaluation on CutBench integrates both automatic and perceptual assessment protocols:

A. Automatic Metrics

• DINO Similarity ($sim_{DINO}$): Cosine similarity between DINO ViT features of the conditioning and generated shots, measuring inter-shot visual continuity.
• CLIP-I Similarity ($sim_{CLIP-I}$): Cosine similarity in the CLIP image embedding space between $S_{cond}$ and $S_{gen}$.
• CLIP-T Fidelity ($sim_{CLIP-T}$): Cosine similarity between the CLIP text embedding of $P^{ind}_{tgt}$ and the CLIP image embedding of the generated shot, assessing text-to-image alignment.
• Fréchet Inception Distance (FID): Quantifies distributional distance between features of generated next shots and ground-truth next shots.

B. Human Perceptual Study

The human study involved 15 raters—9 multimedia graduate students, 3 professional editors, and 3 naïve users—performing side-by-side evaluations on 100 randomly chosen test cases. Each rater assessed model outputs on two criteria:

1. Cinematic Continuity (character and environment consistency)
2. Adherence to Editing (faithful realization of the specified cut type)

No ties were permitted; average preferences and standard deviations were reported (He et al., 11 Aug 2025).

5. Utilization of CutBench for Benchmarking

Next-shot generation models are benchmarked on CutBench by generating next shots conditioned on the same $P^{rel}$0 and hierarchical prompts as provided in the test split. Automatic metrics are computed as described. The “Cut2Next” framework demonstrates significant superiority over the IC-LoRA-Cond baseline across all automatic measures, with, for example, DINO Similarity scores of 0.4952 (Cut2Next) vs. 0.4669 (IC-LoRA-Cond) and FID scores of 59.37 vs. 80.43, respectively.

In human preference studies:

• Cut2Next is preferred 93.7% ± 1.8% for cinematic continuity.
• Cut2Next is preferred 96.5% ± 0.8% for editing adherence.

Case studies highlight typical IC-LoRA-Cond failure modes (e.g., facial mismatches in shot/reverse shot) and ablation findings (e.g., the criticality of relational prompts for honoring edit intent) (He et al., 11 Aug 2025).

6. Reproducibility and Benchmark Protocol

To reproduce the CutBench benchmark:

1. Obtain or construct CuratedCuts by filtering RawCuts to professionally edited shot pairs.
2. Annotate shot pairs with $P^{rel}$1, $P^{rel}$2, and $P^{rel}$3 using Gemini-2.0-flash or equivalent, applying 20% random dropout on detailed attributes within $P^{rel}$4.
3. Split the resulting annotation set into train, validation, and a test subset (holding out several hundred examples), maintaining balance across cut types.
4. Evaluate next-shot models on the test set using the four described automatic metrics; optionally, conduct human perceptual assessment as outlined above.

Each of these steps defines, in aggregate, the CutBench construction and evaluation pipeline as implemented in (He et al., 11 Aug 2025).

7. Context and Significance

CutBench uniquely addresses the need for high-fidelity, cinema-specific evaluation of generative video models, moving beyond conventional metrics that privilege single-frame or purely visual coherence. Its hierarchical annotation schema and professional cut balancing directly target the narrative logic and varied editing conventions found in practical film editing. The benchmark thereby supplies both a rigorous protocol and a dataset foundation for advancing research in next-shot generation, narrative-driven video synthesis, and automatic film editing (He et al., 11 Aug 2025).