ShotWeaver40K: Cinematic Transition Dataset

• ShotWeaver40K is a dataset featuring 40,000 two-shot video clips annotated with shot-level metadata and professional cinematic transitions.
• It employs a multi-stage filtering and annotation pipeline to ensure high quality and narrative continuity in its video clips.
• It facilitates the training and evaluation of controllable multi-shot video generation models with precise editing and cinematographic priors.

ShotWeaver40K is a large-scale, meticulously curated dataset specifically designed for research and benchmarking in controllable multi-shot video generation with professional cinematographic transitions. Developed in conjunction with the ShotDirector framework, it captures the priors and structure of film-like editing patterns by providing 40,000 two-shot video clips annotated with rich shot-level metadata and transition semantics. ShotWeaver40K enables quantitative and qualitative evaluation of generation models, training of edit-pattern-aware architectures, and in-depth empirical analysis of narrative visual storytelling (Wu et al., 11 Dec 2025).

1. Dataset Composition and Annotation Schema

The dataset comprises 40,000 two-shot clips, extracted from an initial pool of approximately 500,000 candidates. These clips distribute evenly across four professional transition types to reflect film-editing priors: cut-in (24%), cut-out (26%), shot/reverse-shot (25%), and multi-angle (15%). Each clip is accompanied by detailed shot-level and global annotations, encoded in per-clip JSON files with the following structure:

• clip_id: A unique string identifier for each clip.
• transition_type: One of {“cut-in”, “cut-out”, “shot/reverse-shot”, “multi-angle”}.
• global_caption: Free-form text summarizing the overall scene or subjects across the two shots.
• shot_captions: An array [caption₁, caption₂] describing the content, cinematographic features (e.g., framing, scale) of each shot.
• camera_intrinsics ($K\in\mathbb{R}^{3\times3}$): Stored as nine floats in row-major order.
• camera_extrinsics ($E_i=[R_i|t_i]$, $R_i\in SO(3)$, $t_i\in\mathbb{R}^3$): For each shot $i$ ($i=1,2$), with shot 1 canonically referenced ($R_1=I$, $t_1=0$).
• Optionally, Plücker ray-maps $P_i\in\mathbb{R}^{H\times W\times 6}$ for each shot, encoding per-pixel geometric information.

Precomputed Plücker embeddings $P_{u,v}=(o\times d_{u,v},\, d_{u,v})$ are included, with $E_i=[R_i|t_i]$0 and $E_i=[R_i|t_i]$1 the camera center.

2. Construction Pipeline

ShotWeaver40K's pipeline integrates multi-stage filtering, annotation, and quality control to ensure both technical and narrative fidelity:

• Raw Source & Preprocessing: Clips originate from 16,000 full-length films. Shot boundaries are detected programmatically using TransNetV2, while similar-segment stitching with ImageBind merges consecutive segments with high visual/semantic similarity.
• Coarse Filtering: Candidates must meet minimum requirements—framerate (≥24fps), resolution (≥720p), temporal duration (5–12s), and an aesthetic score (via LAION-AI aesthetic-predictor) above threshold, with additional scrutiny at each shot boundary.
• Fine-Grained Transition Filtering:
  • Pairs with CLIP-embedding cosine similarity between frames exceeding 0.95 are removed (indicative of no actual transition or flicker artifacts).
  • Pairs lacking semantic continuity are excluded using a vision–language QA filter (Qwen2 prompts).
  • The process ensures retention of exactly two-shot clips per sample.
• Annotation & Quality Control:
  • Hierarchical captions (global and per shot) are generated by GPT-5-mini, encoding both content and editing semantics.
  • Camera poses (extrinsics) are estimated via VGGT, referenced to the first shot.
  • Final validation entails manual spot-checking, automatic verification of caption length, and enforcement of transition-type balance.

3. Dataset Statistics and Empirical Properties

The dataset's quantitative profile, as summarized below, reflects real-world editing priors and variability in visual storytelling:

Transition Type	Count	Percentage
Cut-in	9,600	24%
Cut-out	10,400	26%
Shot/Reverse-shot	10,000	25%
Multi-angle	6,000	15%

Additional summary statistics:

• Average clip duration: 8.72 seconds.
• Average frame resolution: $E_i=[R_i|t_i]$2 (with minor variation).
• Mean aesthetic score: 6.21 (scale 1–10).
• Mean inter-shot CLIP cosine similarity: 0.7817.

Visualization supplements include histograms of durations, aesthetic scores, inter-shot similarity, and qualitative example frames for each transition type.

4. Evaluation Metrics and Editing Priors

ShotWeaver40K supports both dataset analysis and model evaluation through a suite of targeted metrics:

• Shot Continuity ($E_i=[R_i|t_i]$3): $E_i=[R_i|t_i]$4 quantifies visual/semantic continuity between the two shots.
• Transition Confidence Score: For framewise transition logits $E_i=[R_i|t_i]$5, $E_i=[R_i|t_i]$6 (where $E_i=[R_i|t_i]$7 is the sigmoid) measures detection confidence, peaking precisely at hard-cut points in the dataset.
• Transition-Type Prior: Empirical $E_i=[R_i|t_i]$8 encodes how often specific editing patterns occur, aligning with established cinematic grammar.
• Shot Duration Prior: For each clip, the joint distribution of shot lengths $E_i=[R_i|t_i]$9, $R_i\in SO(3)$0 (in frames) is recorded, with durations predominantly in the range $R_i\in SO(3)$1.
• Genre/Scene Diversity: While not explicitly labeled, topic entropy can be estimated via global_captions (e.g., topic modeling or CLIP-embedding clustering).

A plausible implication is that the dataset structure and priors make it well-suited for transfer learning and benchmarking applications demanding explicit control over cinematographic transitions and narrative continuity.

5. Dataset Use, Splitting Protocols, and Accessibility

Standard usage recommends an 80/10/10 split by clip_id, stratified by transition_type, to yield train/validation/test sets representative of editorial pattern diversity. Evaluation, especially in generative settings, uses the ground-truth combination of global_caption, shot_captions, camera intrinsics ($R_i\in SO(3)$2), and extrinsics ($R_i\in SO(3)$3) as conditioning, with primary assessment via the metrics outlined above.

Licensing is under Creative Commons Attribution–NonCommercial (CC BY-NC). The dataset is downloadable at https://uknowsth.github.io/ShotDirector/ (with videos and JSON annotations provided in ZIP archives). Access requires completion of a brief user form. Citing “Wu et al., ShotDirector: ...” is mandated for derivative or comparative work.

6. Context and Research Significance

ShotWeaver40K addresses a persistent gap in existing video-generation corpora: the lack of datasets annotated with explicit, professionally relevant editing patterns and fine-grained 6-DoF camera metadata. By supplying curated two-shot clips spanning major transition types, comprehensive spatial annotation (intrinsics, extrinsics, Plücker embeddings), and hierarchical semantic prompts, it enables directorially controllable generation systems to model both low-level visual consistency and high-level narrative structure.

The dataset underpins the evaluation and training protocol for ShotDirector, but its structural properties and annotation schema make it broadly useful for research in film editing pattern recognition, multi-shot video generation, cinematography analysis, and controllable visual storytelling. Its empirical priors reflect the distribution and sequencing of editing patterns as practiced by human filmmakers, providing a substantive empirical foundation for computational models of narrative video composition (Wu et al., 11 Dec 2025).